This is an impossible question, of course. You can’t know when, unless you are self-publishing. You work to get your ms. in front of agents and publishers, and they decide.
When we started writing K-Town Confidential (Black Rose Writing), a legal thriller with a young female lawyer protagonist, I surveyed the genre to see what the standard was. Aside from John Grisham, who has an engaging voice and other good qualities, the standard was low. We decided to just do what we wanted, and it worked out (2 awards and 4.5 stars average later).
Assuming you have written at least three drafts (and assuming your first draft was heavily cut) and gone a round or two with a professional editor (not your dog-walker), and assuming your work fits into a known commercial genre and is sixty to eighty thousand words in length, you have as good a shot as anyone.
Keep in mind that you want to start your second book before the first one comes out. Why? If the first one doesn’t sell, then it might be bad for your confidence. If the first one is successful, you may feel too much pressure. You want your head to be in your story for #2, not worrying about the fate of #1.
The secret to a decent commercial book is to have an engaging voice. John Grisham has it, though his plots are ponderous. Hermann Koch has it in spades in The Dinner, though I didn’t care for his plot at the end. Philip Kerr in the Berlin Noir series had it up the ying-yang except that he’s dead, and I guess we can’t hold that against him.
Readers will forgive a LOT if your voice is good and you keep it moving.
Raymond Chandler said readers think they care about the plot, but they don’t. They care about the characters. He also said a good mystery is one you would read if the ending were missing.
Zen & the Art of Motorcycle Maintenance was rejected by 127 publishers until William Morrow picked it up for almost nothing. It has sold over 5 million copies. Orwell didn’t think Nineteen Eighty Four would sell (20 million and counting). Franz Kafka on his death bed begged his best friend to burn his manuscripts.
Publishing a novel takes a lot of nerve, you can never be entirely sure. But if you’re not in the game you’ll never know. So get out there and dance.