"'Inspiration' is just a posh word for ideas," Robert Harris tells me, with the easy authority that is his trademark. "You just need a couple of ideas each day to carry you through…" 

I am sitting with the man whom The Telegraph calls "Britain's leading thriller writer" amid the wood-panelled plushness of London's Simpson's-in-the Strand, once a favoured haunt of those great literary entertainers Charles Dickens and Sir Arthur Conan Doyle. My mission is to glean some of the secrets of the writing game from probably the smartest bestselling author at work today. 

Harris's own ideas, however modestly he rates them, have reached millions of readers through his chosen medium of the high-concept, page-turning thriller: "what if Nazi Germany won the war and became a superpower?"; "what if Stalin had a secret son and heir?"; "what if Tony Blair had been recruited by the CIA?"; "what if a self-aware computer attacked the global economy?" The impeccable titles Harris pins on these books are proof of his superior instincts and appeal: Fatherland, Archangel, The Ghost, The Fear Index.

But Harris doesn't do airport lounge wish-fulfilment — no bullet-
proof tough guys pitted against a shadowy "system" — he doesn't do Jack Reacher, in other words. The riveting plots of his novels are informed by a shrewd appreciation of the complex ways in which the world actually works. Readers rely on Harris to guide them confidently into inner sanctums, sites of power, influence and secrets. He learned his trade well enough — from Cambridge to the BBC, The Observer and Sunday Times — but he comes from a Nottingham council estate, and journalism schooled him in how to get his hands dirty with the stuff of real life. His first published books were non-fiction, but they are as expertly assembled and grippingly narrated as the novels.

Occasionally, Harris takes history we thought we knew and transforms it into fiction, as with Pompeii (2003) and An Officer and a Spy (2013), about the Dreyfus Affair political scandal in France (1894–1906). He is obsessed by the special dramas of politics — the shabby compromise, the lesser evil, the dubious means that serve higher ends — and perhaps his most personal endeavour is his trilogy of novels about the Roman orator-statesman Cicero. 

For his latest book he is back in Italy, but for a modern-day story that explores the power, glory and skulduggery behind the process of electing a new pope. "There's something inherently dramatic about an election," he muses. "It's comparable only to an auction or the reading of a will." The name of this papal process, and of Harris's novel, is Conclave.

In person, Harris is a canny, witty, urbane man. Our conversation is interrupted briefly by another Simpson's patron who wants to give Harris his regards, but not for one of the novels. It transpires this chap has been keenly following Harris's Twitter campaign urging people to register to vote in the upcoming Labour Party leadership election. A relative newcomer to Twitter, Harris has swiftly mastered the platform, his clarity of thought and punchiness of phrase are perfectly displayed in 140 characters. But our chat is about his longer-form accomplishments: about Conclave and, more broadly, the path he has taken, the craft he has perfected, in the writing of bestselling fiction.

ESQUIRE: What lured you into the Vatican as a book setting?

ROBERT HARRIS: "My main preoccupations as a writer have been with power and history, and the two are almost perfectly fused in how popes are elected. As I was researching it, it just felt to me like a perfect story. You have 118 cardinal-electors coming to the Vatican from all over the globe to cast their votes, they're locked up in the Sistine Chapel and there's a series of ballots, at the end of which one among these men becomes pope. Outside, the whole world is waiting for their decision, particularly the world's one-and-a-quarter-billion Catholics. And it all unfolds over 72 hours, within a space of something like 500sq yards. So the idea of going inside the Sistine Chapel after the doors slam shut, and describing what happens… the journalist in me just loved that."

ESQ: Guiding readers into an enclosed, powerful, secretive world seems to have become a signature of your work.

RH: "Selling Hitler (1986), my biggest non-fiction book, was about a group of men who became obsessed with the belief that they had gotten hold of Hitler's secret diaries, and they were almost literally locked in a room with those diaries. That book contains almost everything I've written about since. Group dynamics, power, men in rooms going crazy. It's all there, really."

ESQ: Writers are very often advised to "write what they know", from their own experience, yet you very conspicuously haven't done that.

RH: "Yes, I wasn't a Nazi, I wasn't a Bletchley codebreaker. My first novel wasn't published until I was 35, which is getting on in life. But if I'd started sooner I wouldn't have had the material, I wouldn't have travelled enough, or met enough interesting people. What I had, from an early age, was a passion for history and politics and ideas. I had an unhealthy interest in World War II and Nazi Germany, in a way, that was 'what I knew.' So, just as my brother-in-law [Nick Hornby] writes about football, I've written about Nazis. Each to his own obsession. Perhaps the advice ought to be, 'write what you feel passionate about.'"

ESQ: Having acquired so much experience in non-fiction, was there a moment you knew you wanted to leap into making things up instead?

RH: "Selling Hitler had banged at the window of fiction. Fatherland took me through it, and I never went back. It was the summer of 1987, I was a journalist for the BBC, travelling in Germany a lot for a Panorama story. I took my then-girlfriend, now wife, to Sicily on holiday, and I was swimming at the beach one day and surfaced to hear all these German voices around me. And quite suddenly I thought, 'This is exactly what the world would have been like if Germany had won the war. They would have been on all the beaches, efficient Lufthansa flights everywhere…' The plot of Fatherland just floated into my head: Germany in the Sixties, 20 years after winning the war, the economically dominant power in Europe. So I wallowed about in the water for another 20 minutes, then I swam in and tried to find a pen and paper."

ESQ: Did the actual writing of Fatherland come as easily as the eureka moment of inspiration?

RH: "No, it took me three years to produce a manuscript, and there were moments when I was absolutely certain I would never manage it. I had the classic experience of a first-time novelist. I wrote the first few pages perfectly happily, got my characters together in one room at Berlin police headquarters, then the book just ground to a halt. I put it away and did other things. But my wife had given up work, we had a child, a huge mortgage, I had a newspaper column which was half the money we needed to live on, but the other half had to come from me writing a book a year…."

ESQ: So you knew you had to finish it. How did you get out of that rut?

RH: "My agent sent me an essay by John Irving, who said that in writing a novel you have to know the end before you begin. Otherwise the book is a fraud and a lie. Novels are like news stories one can approach as a journalist — a crash, say, or a scandal. The events have already occurred and your job is to tell the story. For me, that transformed my material. I wrote the first third of Fatherland and sent it to my American agent. Two weeks later he said he had 12 publishers interested. There was an auction, it sold for more money than I'd earned in all my life, and everything changed."

ESQ: With that success, did you feel you were off and away into your new profession?

RH: "Actually, it was a tough period that followed with Enigma (1995). The line about the 'difficult second novel' is right. A lot of people have one story in them but with the second one you've got to try painfully to do it all again. Fatherland hadn't been based on any reservoir of technique. I was like a tennis player who'd managed to take down a champion without having learned to serve and volley. But you have to develop a game and proper strokes, you can't just fluke it. And Enigma wasn't even the end of that. It took me four novels, and all four were absolute bastards to write."

ESQ: So how long did it take you to feel like a fully qualified novelist?

RH: "With Pompeii (2003). I was convinced it would be a disaster, in fact, it was the one where I thought, 'OK, I can do this now.' It was September 2002, I had given up journalism, it was a horrible grey day and I was trying to solve a difficult problem in the novel. But, I realised, I was happy, happier in the misery of novel-writing than I would have been successfully writing some piece of journalism. It was like trying to solve a chess puzzle, finding pleasure in the pain of it. And I was fine thereafter."

ESQ: How much stress do you put on finding the right beginning for a novel?

RH: "I really believe that 50 per cent of the whole effort of writing a novel is in the first paragraph. Essentially, it is like telling a joke, a matter of hooking your audience in from the start. 'Three men walk into a bar' isn't a bad opening line, ever. Once you've got the first paragraph right — if you've picked the right situation, the right point of view to describe it from — then you're fine, and it becomes fun. But you're very lucky if you get that.

"A lot of my books I've begun and then realised that's not where the story should start. Fatherland I planned to open with a television broadcast showing the first men landing on the moon, planting a swastika there… which isn't a bad start, but it wasn't right, it belonged to a different book. Fatherland needed to be a grimy police noir, not science fiction. 

"The thing is, nobody embarks on a joke without knowing the punchline, which is why I think you have to be the master of what you're going to tell before you start. Not too much — books can't follow a rigid grid — otherwise it becomes boring to write every day. You need to be surprised by what happens when characters become more rounded and new possibilities occur… It's a bit like flying a glider. You're up there, the journey's pleasant, and you hope, broadly speaking, you'll be able to put it down. But you're not necessarily certain what field you'll be landing in."

ESQ: So, when it comes to endings, do you always know in advance? Or do you ever surprise yourself when you get to the last page?

RH: "I didn't know the final twist in The Ghost (2007) until quite a way through the writing. And the twist in Conclave, I wasn't sure I'd have the nerve to write it until I got quite near the end. There was a more conventional way to do it, and I did think that was what I'd do. But, instead…"

ESQ: The protagonists of your books are often complicated men: in Fatherland, say, the Nazi policeman who has an uneasy conscience, or the brilliant but neurotic codebreaker in Enigma. Do you need to find some element of yourself in your hero? 

RH: "In the end, every writer writes about themselves and their preoccupations. I like my central characters to be full of doubts and contradictions, I'm not one for great macho heroes. But my job as a novelist, also the great pleasure, is to try to put myself inside the heads of people who are nothing like me, be it codebreakers, Roman senators or French army officers. With Conclave, I knew the most interesting character would be the dean of the college of cardinals, an elderly celibate whose responsibility is running the papal election. Trying to get inside that man's head became another challenge."

ESQ: Is it vital your readers find your lead characters sympathetic? 

RH: "I think if the reader's going to spend time with them, they have to find them likeable. The only book of mine that has a slightly unsympathetic figure is The Fear Index (2011), and a lot of readers didn't like that — it was noticeable. I was trying to be honest, and a 'quant' of that kind [quantitative analyst, ie, a maths whiz who designs computer models to help hedge funds bet on the markets] is a pretty cold fish; he's not a villain, just rather remote from the rest of us. But then that book was an attempt to rewrite Frankenstein, and Victor Frankenstein is also not a very attractive figure, yet that's still a successful book."

ESQ: The action in The Fear Index unfolds over 24 hours, even fewer than Conclave. Fatherland and Enigma both take place over just four days. Is this kind of pacing important to you?

RH: "Yes, my advice for a thriller writer is to choose a tight time-frame, and then shorten it. Just ruthlessly cut any stuff that isn't necessary.
I don't always do it; the Cicero trilogy is set over 25 years. But my ideal would be to write a book almost in real time. For me, the power of the novel is the human drama of elapsing time that you can feel around you. In life, we're constantly caught in the great flux of some story like, 'Who's going to win this election?' And I constantly want to know what's going to happen next."

ESQ: You now write a novel a year. Are you a stickler for regular work routines? Or is your process a bit more laissez-faire? 

RH: "I don't understand writers who work through the night, because I'm utterly exhausted by then. I get up quite early, I start by going back over what I did before. And, by and large, my working day finishes around lunchtime, then I go and do other things. But I do believe in what Stephen King has said about the value of starting to write early in the morning, while your brain is still in that sleepy, half-dreamy state where things are plastic. You're willing to trust your subconscious, and the night thoughts that can linger on into the first minutes of the day. Stephen King calls them 'the boys in the basement' who do the real work on a book. Very often I find problems get solved that way, things get knitted together. And it's happened while I've not been thinking about the book. I've been asleep, or walking the dog, whatever."

ESQ: Another thing fledgling writers are forever told is to ditch any ideas they can't get to work, to "kill their children" in the frightening parlance. Are you that ruthless?

RH: "Actually, my number one rule is, 'Never throw anything away.' I've learned that material can come back and be used years later. After Archangel in 1998, there was a five-year gap in my résumé,
during which I had lots of ideas that I gave up after a year or so. One was an idea for a play about a ghost writer and an ex-famous person, but I couldn't make it work. A few years passed and then I heard on the radio something about Tony Blair maybe having to go to live in America in case he might be prosecuted for war crimes, and suddenly my old idea resurfaced: an amoral ghostwriter for hire, who falls into a world of lunatics and power-crazed, corrupt individuals. I loved writing The Ghost, it was the easiest of all my books, took me three months. But it had been in the back of my mind
a long time."

ESQ: Your novels have been successfully adapted for movies and
television, most prominently Roman Polanski's 2010 film of The Ghost. Did working with such a genius of cinema teach you anything about novel-writing?

RH: "Roman is very good on pure storytelling, and his greatest movies — Chinatown being the pinnacle — are stories that illuminate something much bigger. We have a shared love of reaching an audience, and a disdain for what he calls 'art-house schmart-house'. He's very rigorous, very sharp-eyed. I actually sent him the manuscript of An Officer and a Spy (2013) and he read it with immense care, then called me from Biarritz and gave me 300 notes down the phone; it must have taken three hours. He started off, 'OK, page one, "I go in to see two generals in a room, they are standing warming their backsides by the fire, they turn to look at me…" Are they coming down the fucking chimney or what?' That's the ruthless eye of a director."

ESQ: Speaking of directors, I've heard you mention elsewhere that as a novelist you have a special admiration for Stanley Kubrick?

RH: "I certainly don't seek to compare myself. But I do really admire that tremendous thing Kubrick could do, to make stories set in the future [2001: A Space Odyssey, 1968; A Clockwork Orange, 1971], or World War I [Paths of Glory, 1957], or the Roman era [Spartacus, 1960], or the 18th century [Barry Lyndon, 1975]. Each of them is 'a Kubrick film', and they're about his imagination, really, rather than just the genre. It was with Pompeii that I realised if you can break away from writing about your own time then you can write about anything in the world. You can set a story in the Middle Ages or the future, it wouldn't matter, it would just be a story. And what that means is I'm not going to run out of subject matter in the way other writers might. I've enjoyed being in Ancient Rome and 19th-century France and the Vatican. With each book I feel like I've slipped into that world, and it's been a pleasure, like being a child, really. I feel like a grown man who's paid to be a child."