t has become quite the fashion for contemporary writers to have a go at reinventing past classics. We've seen it done with imperishable literary heroes like Bond, Jeeves, Poirot and Winnie-the-Pooh. This is a relatively new phenomenon. But not in the case of Sherlock Holmes, where the practice is almost as old as the man himself.

Sherlock Holmes was the conception of Arthur Conan Doyle, a Scottish doctor down on his luck who had turned his hand to fiction to make up for a worrying lack of success in attracting patients to his surgery. Conan Doyle could never have guessed that the first Sherlock Holmes story he wrote, A Study in Scarlet, published in the 1887 issue of Beeton's Christmas Annual, would in a few years propel him and the characters he created to literary fame. The personalities and characteristics of Holmes and Watson became as familiar to late Victorians as their own friends and families. The readership was worldwide and spread across all ages and classes.

Conan Doyle himself – frustrated by the burdensome discomfort of what he felt had become something of a millstone around his neck – killed off his creation by having Holmes apparently plummet to his death in Switzerland's Reichenbach Falls, pulling his arch-enemy Moriarty down with him into the roaring cataract. The public outcry was swift and over­whelming. Grown men and women wept openly, black armbands were adopted, and a host of angry readers wrote abusive letters to Conan Doyle's publishers, to the Strand magazine and to Conan Doyle himself. He grudgingly bowed to public pressure and unearthed a story set before Holmes's plunge into the abyss – The Hound of the Baskervilles. But only after the clamour still refused to subside did he at last bring his hero back to life in The Return of Sherlock Holmes, presaging a new stream of short stories and a full-length novel, The Valley of Fear. He was to continue writing Sherlock Holmes adventures right up until 1927, although he never set a story later than the eve of the first world war.

During his lifetime, stage productions (he collaborated on the first himself) and silent films were produced, and after his death in 1930 the great post-Conan Doyle Holmesian caravan began to rumble into full life.

Now, you should know that the Sherlock Holmes body of work written by Arthur Conan Doyle himself is called by Sherlockians "the Canon" (coincidentally and happily an anagram of Conan). The Canon comprises four full-length novels and five collections of short stories.

In all, 60 canonical stories. How many have been written since? I don't believe it is possible to count. I know I have written one myself. Conan Doyle's son Adrian Conan Doyle, along with the great Golden Age mystery writer John Dickson Carr, wrote a collection of new stories under the title The Exploits of Sherlock Holmes. After that, the stampede was on, in all media, of course. The Hollywood films starring Basil Rathbone as Holmes and Nigel Bruce as Watson began in 1939 with the canonical The Hound of the Baskervilles, but soon began to move the pair into all kinds of modern settings. By the 1940s, they were doing their bit for the homeland by pitting themselves against the Nazis. The Conan Doyle Estate was open to allowing and authorising such variations, which soon became an established and legitimised part of the whole Sherlock Holmes enterprise. Between 2014 and 2023 Holmes emerged from copyright and entered the public domain.

Most Sherlockians I have come across will, like me, cite Jeremy Brett as their favourite Holmes, but one feels there will always be a new kind of Sherlock for every age. I had the honour of playing Sherlock's brother Mycroft in Guy Ritchie's Sherlock Holmes: A Game of Shadows, with Robert Downey Jr and Jude Law as Holmes and Watson, and the pleasure of reading the entire Canon for Audible. Along with the publication of my own Holmes pastiche, these were the completion of some kind of a lifelong dream. The closing of a very strange circle.

I can say without the slightest fear of exaggeration that Sherlock Holmes altered and determined the whole course of my life. Let me explain.

I had been entranced by the Sherlock Holmes stories from the earliest age. I read and reread the entire corpus of stories avidly and before long I had joined, as their youngest member to date, the august Sherlock Holmes Society of London. This eccentrically aristocratic institution had Lord Gore-Booth as chair and the Marquess of Donegal as editor of its magazine, the Sherlock Holmes Journal. I managed to get permission from my housemaster to leave school one weekend, travel by train to London and attend a society meeting, at which I was scheduled to read a paper. The subject of my disquisition? Well, I had noticed with excitement one day the parallels between Sherlock Holmes's description of Professor James Moriarty ("He is extremely tall and thin, his forehead domes out in a white curve, and his two eyes are deeply sunken in his head") and TS Eliot's description of Macavity the Mystery Cat ("his eyes are sunken in. His brow is deeply lined with thought, his head is highly domed"). The society was being very kind in letting me address them on this topic: any fool could see that Eliot had based his Macavity entirely on Moriarty, it was hardly news. I thought I had discovered a great literary secret. At any rate, the event went well, but the sequel did not.

I stayed in London, sitting in cinemas from dawn till midnight and watching any number of films. By the time I came to and hopped on a train back to school, a full three days later than had been agreed and sanctioned, I found that I had been officially declared missing. My parents and the police were informed, and the hunt was on. The fat was truly in the fire, my number finally up. The school expelled me. That expulsion led to other expulsions and disasters, which culminated in my landing in prison; I shan't go into too much detail: Moab Is My Washpot, the first volume of my autobiography, is also happily still in print, and the full story can be read there.

Anyway, all these ructions finally got me thinking straight and determining to change my ways and try for Cambridge, the university on which I had always set my heart. I did get there, where I met my friend Hugh Laurie and found myself set on the course of writing and performing which has filled my life ever since. None of which could have happened without that Sherlock Holmes Society evening and its cascade of consequences back in the early 70s.

But I still haven't quite addressed the most pertinent question. ? Why do these characters – Holmes, Watson, Mrs Hudson and Mycroft (along with that hapless cast of professional bunglers, sceptics and sincere admirers from Scotland Yard) – seem to attract reinvention and retelling so much more than any other fictional creations? And why the Game? How did that arise? The Game, I should explain, is what Sherlockians call the tradition of Holmesian scholarship that makes the very serious and straight-faced assumption that Holmes and Watson were real and that they actually did live at 221B Baker Street and that everything Watson writes about (although he makes plenty of mistakes in consistency and continuity) truly did historically happen.

But I repeat: ? This doesn't happen with Poirot, Father Brown or Lord Peter Wimsey. What's so special about the denizens of Baker Street?

Well, one can begin by making the surely undeniable statement that Sherlock Holmes is one of the most fully realised and enduring fictional characters ever created. Very quickly, his complex and rounded personality, the details of his intellect, his manners, his dress, his habits, his appetites, techniques and methods combined to create a place for him in the public consciousness of a kind that had never before been experienced in literature.

Holmes's world is also vivid and tangible. Everyone who has read even a handful of stories will have their own clear picture of the Baker Street rooms, the fireplace, the test tubes, the violin case, the pipe rack, the fog at the window … we all have our images, reaffirmed or modified by countless films and television adaptations certainly, but unmistakably real. So real, indeed, that even as the stories were still being written and published, tens of thousands around the world behaved as if Holmes and Watson truly lived and breathed. From as far as Japan and South America came letters addressed to 221B Baker Street begging for Holmes's assistance. Some even wrote to Buckingham Palace and Downing Street to urge the king and prime minister to award Sherlock Holmes a knighthood.

But there is, I think, another reason for the astonishing connection that Holmes made with the public on his arrival and that he continues to make in all the multiplicity of his incarnations today. And that is need. The world needed him in 1887 and the world needs him now.

With the great rise in literacy from the middle of the 19th century onwards, it perhaps seems inevitable that a new kind of fiction should arise that tried to solve the puzzle of the lately arrived Big City – most especially the biggest by far in the world, London. The daunting size of the metropolis was something quite new in human history. Society was transforming itself from a knowable hierarchy into a bewildering network. It could be seen structurally as like a great web, interlinked and complex, mastered only by those sensitive enough to understand, touch, feel and identify every node and every twitch on the filament. In fact, by a Sherlock Holmes.

And after Holmes, of course, ten thousand more detectives sprang up, all with their own USPs, their own special ways of clearing up mystery and caging the monsters. But so, too, did a strange breed of 20th-century heroes who offered a similar promise of bringing light and order to the darkness and chaos of modernity, but in their case without bothering to think much. I mean, of course, the comic book and film superheroes. Our infantile side (and there was never an age in our history that so encouraged and coddled our inner child as the present) seems to have embraced magic, flying, invisibility, X-ray vision and all kinds of impossible superpowers. It seems to me fair to suggest that Superman and Batman, unlikely as it may seem, follow in the great detective's footsteps and owe him their being. He was surely the first caped crusader, for all that his cape was constructed of good tweed. They, like him, answered a need.

Why so many have moved from valuing the real and achievable powers that Holmes exhibits – achieved through his own self-training, study and practice – to valuing imaginary, unearned, randomly achieved magical superpowers is another question. But fortunately the appetite for Holmes and real deductive talent is undiminished, too. We can have both kinds of hero, after all. On television alone, for every Superman and Deadpool, there has always been – as well as a new Sherlock for each new generation – a Columbo, a Monk or a Ludwig reaffirming the power of the Sherlockian method and re‑establishing the primacy of mind over mayhem. Holmes's cigar ash, footprints and almost invisible telltale particles are the forerunners of the crime lab's mitochondrial DNA, epithelial particles and microfibres, and his magnifying glass gives way to today's scanning electron microscope. But the principles are quite unchanged. Which is why he is the presiding spirit behind every CSI operative and why the modern Scotland Yard criminal database (the Home Office Large Major Enquiry System) has initials that spell out the name HOLMES.

Like London and the wider human world, the world of Holmes and Watson is a work in progress, capable of being modernised and made over in countless ways, but always, underneath it all, entirely its noble and magnificent self.