This is a Disclaimer to those picky types who I know will call me out on it - THESE NEXT FEW ARTICLES WILL CONTAIN ANALYSIS OF THE ENDINGS OF NOVELS. IF YOU FEEL THIS WOULD LESSEN YOUR ENJOYMENT OF THEM, THEN DO NOT READ ON.
Now I've got that out of the way, I'd like to turn our attention to the title. The ending of a novel should in some way provide a logical and natural conclusion to the plot. Therefore, the first criterion of a good ending must be:
1) To Logically Resolve What Has Gone Before.
While the Ancient Greeks were happy to introduce a Deus Ex Machina if they'd written themselves into a hole (God from the Machine - in essence, Zeus would come in and say something along the lines of "This isn't right - let's reassess the situation"), this doesn't hold as much weight nowadays. It's called lazy plotting, and it's generally a sign of bad sci-fi/horror/adventure. Unstoppable villain about to win? Here's a superweapon out of nowhere.
Logical resolution of a plot must therefore avoid the introduction of forces outside the characters' control, unless this can be rationalised. Take a war story for example. It's a nice ending if all of a sudden ten tanks arrive and destroy the five threatening your heroes, and it's not as bad as "Jesus comes along and destroys the tanks with JESUS LASERS" but it doesn't make for a good ending unless the heroes have in some way organised this (if it's part of the plan, then go ahead.)
Perhaps a better ending would therefore have the heroes win with cunning and skill? Possibly. But the "work a solution out of nowhere" doesn't always work. For example, if the "cunning and skill" simply involves getting the better of an incompetent enemy with traps et al, unless they're characterful they won't really work. Guerilla warfare may be an option, but having five men take on overwhelming odds with only a shoestring armoury and some sticky-back plastic is really quite weak and will ruin any suspension of disbelief that your readers may have had.
If our first rule of an ending is "Resolve the Plot" then does a book which doesn't do this fail as a work of fiction? Of course not. Life doesn't always resolve nicely, and if you're in a situation where you as an author cannot work out an ending which doesn't pass Rule One then leaving it open can work. It allows the reader to imagine the ending.
The author Iain M. Banks is very good at this. Take, for example, his work "Feersum Endjinn." The essential plot is that the protagonists seek an object which will avert an impending crisis. However, the ending doesn't make completely clear a) what they found, b) if it worked and c) what happened next. Surely this fails Rule One - we don't know if the crisis was averted? However, Banks has evidently taken the decision to leave it to the reader since he did not feel capable of resolving events in print in a satisfactory way. We are given some hints, but the ultimate decision is left to us.
In fact, even a logical resolution of the plot can be weaker than an open ending. A technique prevalent in the Victorian era was to end with a wedding, either of your protagonist (Esther marrying Mr. Jarndyce in "Bleak House") or where your protagonist is involved (Clym's pensive fate at the end of "The Return of the Native.") While this is fine most of the time, it can seem a bit twee and overdone (a wedding is a new beginning, marking the end of the plot &c &c). Either end strongly in a way that will reassure your audience that the adventure is over (while avoiding a cliche or Deus Ex Machina) have an open ending (but only if it's necessary - I'll come on the downsides of this next!)
My word, ending your novel is looking hard! You shouldn't end on a cliche, you should know when to stop (the wholly unnecessary ending of the film adaptation of "The Return of the King" is testament to that) and when to carry on! But I'm no expert. These rules are merely observations and guidelines.
So an open ending helps you avoid all the cliches of a concrete one. Result! But not so fast. Take the ending of "The Dark Tower" by Stephen King. He in fact provided two endings. An open one, then a disclaimer saying "IF YOU DON'T WANT YOUR EXPERIENCE SPOILED, DO NOT READ ON!" and then a curious twist ending. Interesting, to be sure. But a sign that he himself had doubts about how to end his magnum opus (if seven long novels don't count as one, I don't know what does). It would have been better to either write a good conclusion (not a twist ending) or a good open ending rather than making a half-hearted attempt at both.
The temptation was always there for me to read on, whereas if the option for closure had not been there, I would have been happy (and thought that King had real cojones for ending an epic tale of a journey with
no description of what was at it's end.)
Dystopic novels like open endings. In "1984" we are left with Winston "loving Big Brother" while in "Brave New World" the Savage is dead but from there, what next? These seem fairly concrete. The plot is resolved, after all. But the open-endedness still exists. Will things change? This device is, however, swiftly becoming the new lazy writing. Far easier, after all, to leave them hanging on your last word! It's my opinion, therefore that an open ending in fact obey Rule One. It
logically resolves the plot. As to what happens next? Well, that's a whole new story.
I'll call it a day now, but join me later for more exciting endings.
Including:
-What if it never happened?
-And that's how I got here!
And...
-The killer is INSIDE THE HOUSE!