he email from Sotheby's might easily have gone unread. Despite his deep appreciation for the musical heroes who came before him, Gary Kemp is not a collector of rock memorabilia. But there, among the auction lots, was something that caught his eye: a 160x100mm black Twinlock binder that once belonged to the Beatles' manager, Brian Epstein.

The listing detailed 57 pages containing 404 entries, many handwritten by Epstein himself, cataloguing the social and business network of the Liverpudlian impresario in 1967, the year of his death. Kemp immediately recognised this was much more than simply an address book: it was a who's who of 60s London at its swinging peak, from David Bailey and David Frost to most of the Rolling Stones. "I just thought, 'I've got to have this. I've got to hold it,'" he says. "It's got DNA on it."

On the day of the auction, Kemp, the songwriter and guitarist of Spandau Ballet, was in a recording studio bidding by phone. As the price quickly soared, so too did his determination. "I'm still pressing the yes button," he remembers, "thinking, 'My wife's going to kill me for this!'"

After bidding £10,710, Kemp became the book's fifth owner. After Epstein's death, it had passed to his brother, then his nephew, and then one more sale before reaching the auction at Sotheby's. For Kemp, owning it felt like the culmination of a lifelong fascination with the Beatles and a long-held admiration for Epstein, the man who discovered the group at The Cavern in late 1961.

Kemp is talking from his study in an elegant five-storey Georgian townhouse he attributes, in part, to the inspiration and opportunities the Beatles gave him. "If it hadn't been for Paul meeting John, I wouldn't be living in this house," he says. "But really, if it hadn't been for Brian walking down those steps that November lunchtime, I wouldn't be in this house either."

One visitor to Kemp's home shortly after he'd acquired Epstein's address book was art film-maker Simon Halfon, a friend since the 90s. He was just leaving when Kemp thought to show him his latest purchase. "I was obviously thrilled to look at it," says Halfon. "An address book in many ways is a very personal thing. It's not a diary, but it tells you who your friends are, your family, your business acquaintances, the places you hang out. It's quite telling of an individual's personality."

Halfon was an auspicious person to show it to. He had spent years fascinated by Epstein, developing a biopic of the manager with Jude Law attached to star, but it never came to fruition. Now, confronted with Epstein's address book, and the revelatory list of contacts within, he saw a different story waiting to be told. "I called him on the way home," says Halfon. "I said, 'Christ, Gary, this is a book.' And he was up for it straight away."

"I realised," says Kemp, "the only way to do this is to tell Brian's story through his contacts. This was Brian 360."

What followed was an exhaustive piece of detective work. Name by name, Kemp scoured books, newspapers, magazines and online archives, trying to establish not only who each person was, but why they had crossed paths with Epstein. Meanwhile, Halfon was tasked with sourcing corresponding imagery. Some names proved straightforward, while others were little more than shadows they'd futilely chase.

The result is 1967: The World of Brian Epstein and the Beatles – not a biography in the traditional sense, but an alphabetical list of fragments that, read in sequence, reveals an unexpectedly intimate picture of a complex man unravelling in the last months of his life. Through the people he knew, it captures Epstein juggling the increasingly elaborate business affairs of the Beatles within an international set of artists, managers and lawyers, while wrestling with the more tormenting aspects of his own life. And because these are the people who surrounded him in 1967, the book also depicts the circumstances closing in around him.

"It was really a key year for Brian," Kemp explains. "His contract is up with the Beatles. And the Beatles have stopped playing live, which gives him less purpose."

This sense of unease permeates the book. Struggling to keep pace with his expanding empire, that January Epstein merged his NEMS Enterprises with Robert Stigwood's agency to ease the workload, causing friction with the Beatles. "He's terrified," says Kemp. "He finds these boys who he transforms into the biggest phenomenon the world has seen, but then it starts to run away from him. And we're at this stage now where he is panicked. He can't sleep. He's taking uppers and downers."

Epstein's drug use exacerbated an already precarious mental state. The book traces his encounters with addiction doctors, the gambling dens he frequented, and his stints in rehab at The Priory in London. His Jewishness often made him feel like an outsider, and he was gay at a time when it was still a criminal offence – part of the complicated landscape he privately navigated.

"There's such a strong sense of the gay community within the book," says Kemp, "whether it's Brian's lawyer, David Jacobs, the people he brought down from Liverpool … or people in the book, like Robin Maugham, Somerset Maugham's nephew, who all had gay relationships that were quite open in that they lived together."

What's striking about this book is that despite reconstructing the remarkable breadth of Brian's orbit, he emerges as a lonely, isolated figure. "He couldn't find a relationship," says Kemp, who uncovers in these pages a handful of likely liaisons – men who passed briefly through Brian's life.

The most disturbing of these is John "Dizz" Gillespie, an aspiring American actor with whom Brian had a volatile, on-off affair. The last time they'd met, in August 1966, Dizz had stolen Brian's attache case from his Los Angeles hotel room, making off with $20,000, contracts, incriminating photographs, and – perhaps – his previous address book. The betrayal, distress and persistent threat of blackmail and/or exposure sent Brian into a downward spiral.

It is startling, therefore, to find Dizz's name here at all, whether Brian included it out of guilt, desperation or lingering hope. Gillespie left almost no trace of himself. "There's not even a mugshot," says Halfon, "which you'd think there would be, because he was a bit of a villain in his own way." Finally, Halfon discovered a signed copy of John Lennon's A Spaniard in the Works, inscribed to Dizz with a characteristically acerbic note. "That was at least something," he continues, "to prove his existence and interweave it into Brian's world."

Epstein was alone when he died, aged just 32, of an accidental overdose of sleeping pills, on 27 August. He'd spent the bank holiday weekend at his country house in Sussex, but a guest he'd hoped to see failed to show, and a disappointed, drunk Epstein drove home to London that same night, locking himself in his Chapel Street bedroom.

Kemp imagines Epstein there, withdrawn and despondent, reaching for his address book in need of friendly reassurance. "It was almost certainly on his bed when he was found dead," Kemp says, "because he was calling people, and he would have been calling people using that book."

The reward for months of scrutiny into Epstein's short life was, for Kemp, a closeness with the man himself. "I came out knowing him more. I felt I'd walked with him down different streets," he says. "Because the book has addresses in it, you see it geographically as well."

Those locations are vital in bringing the physical world Epstein inhabited so vividly alive in the book. London sparkles with references to its clubs, offices, restaurants and homes, while further afield, the feverish nightclub culture of New York and the glamorous Hollywood scene (via Dean Martin, Colonel Tom Parker, Orson Welles) crackle with the same electricity. Epstein feels present throughout, fluttering through the worlds of music, theatre, business and bohemia, constantly making new connections along the way.

Together, those details create an evocative, three-dimensional portrait of a man often overshadowed by the towering legends of his most famous charges. "It'd just be nice to push Brian slightly to the foreground," says Halfon. "It's very hard to stand in front of the Beatles in a way, as a character, because they're so larger than life. But people can say, 'Well, this was the guy who made their dream come true.'"

But what of the Fab Four themselves? Anyone flicking through the pages expecting to spot John, Paul, George or Ringo's names will be sorely disappointed. Their exclusion, Kemp suggests in the introduction, may be down to security reasons – particularly in light of Dizz Gillespie's thieving. But appearances, as readers will find out, can be deceiving.

It's been a voyage of intrigue that Kemp seems reluctant to end. "Maybe I've started a thing now," he laughs. "Frank Sinatra's got an address book out there somewhere."

1967: The World of Brian Epstein and the Beatles by Gary Kemp and Simon Halfon (HarperCollins, £25) is published on 13 October