ix years ago, Tim Walker was walking in the Highbury Fields area of London when he came across a plaque on the side of a building. In 1970, at these now-converted public toilets, Louis Eakes, chairman of the Young Liberals, was arrested for importuning following a "pretty police" entrapment in which male officers posed as would-be sex partners.

It was here, the plaque told the photographer, that the first ever gay rights demonstration in Britain had taken place, organised by the Gay Liberation Front, in protest at Eakes's arrest. But Walker was jolted by the date: 27 November 1970. It was the day he was born. So he wondered: "Who were those people out marching that day? Where are they now?"

This was the genesis of Walker's new exhibition, Fairyland: Love and Legends, about to open at the National Portrait Gallery in London. Its rooms of photographic portraits represent a tapestry of queer community in Britain spanning decades and taking in activists, artists, writers, models and everyday heroes, from nurses to physicians and army veterans.

The fairy concept was inspired by Walker's childhood. "My brother and I used to share a bedroom, and over our beds was a scene of fairyland, with little pixies and elves. I was always making camps in the garden. And by the time I was sent to school, the word fairy was derogatory. So I think the fantasy and escapism got muddied by the bullying." The exhibition, then, is a long-overdue self-affirmation.

I meet Walker at his studio in Shoreditch. Gentle and attentive, he serves peppermint tea, ginger biscuits and gluten-free jaffa cakes on a tray of china and strolls around a scale model of the exhibition as he talks me through it. Of course, it starts with the gay men and women on the Highbury march: Ted Brown, Lisa Power and Andrew Lumsden. Power and Lumsden are shot to mimic the Northern Renaissance portrait style of Hans Holbein the Younger. "You're in the National Portrait Gallery," Walker explains. "If you go to the top floor, you have all these Holbein paintings. I wanted to put the people at the march into that realm of Holbein-esque celebration."

Not everyone desired to be dressed up in ruffles and French hoods, so activists Peter Tatchell, Phyllis Opoku-Gyimah (Lady Phyll) and Sue Sanders are shot against Walker's trademark white background, as are queer stars including Ian McKellen, Troye Sivan and Chappell Roan. Why this style? "That's the cheapest way," Walker says. His method is collaborative; he discussed with each person how they'd like to be shot and asked them to bring significant items. Russell T Davies is covered in pages from the script for his drama It's a Sin, while photographer Sunil Gupta brings a camera.

In the show's most arresting moment, Graham McKerrow is depicted in a pork pie hat and suit, holding a November 1982 copy of Capital Gay, the newspaper he co-founded with the late Michael Mason. The splash says: "US disease hits London." This, says Walker, "really moved me, because he brought this newspaper from when the first news hit about Aids. That was the first time anyone was talking about it here."

That history of Aids runs through the exhibition. In one emotional image, nurse Jane Bruton, who has worked with HIV patients for 40 years, is depicted holding her former patient Tony Whitehead, one of the first people whose lives were saved by combination therapy in 1996. They are photographed with Jonathan Weber, a professor of medicine who was part of that scientific breakthrough. "It was such a beautiful reunion," Walker says, "but what really touched me was their gestures: that care and pride was still alive."

Capturing this was important to Walker because he remembered sitting in the living room as a teenager with his family watching the falling tombstone from the Don't Die of Ignorance public health TV campaign launched in 1987 by health secretary Norman Fowler to warn the public about HIV and Aids. "I feel that set the whole gay liberation movement back, it was such a terrible thing."

After the white backgrounds, the fantasy portraits of fairies, floral teapots and Meissen candlestick holders, the most explicit photography comes at the exhibition's close: erotic lesbian shots, a surrealist image of butt cheeks impressed on a goldfish bowl, a giant rat and octopuses strewn across submitting models and an S&M dalmatian pup suspended in the air. "There's kinky play where people dress up as dogs," Walker notes. "I think there's worlds here, erotic sexual landscapes I call them."

The backgrounds to these images are baroque, Walker having been inspired by Jean-Honoré Fragonard's oil painting The Swing in London's Wallace Collection. He originally conceived of these more risque images being scattered throughout the exhibition. But the National Portrait Gallery wouldn't have it: "They wanted to push everything to the back so that could be an 18-plus room," he says, scoffing.

Walker was born in Guildford; his mother wrote cookbooks and his father worked in industry. He attended Charterhouse boarding school which he describes as "horrific, absolutely Victorian, so homophobic". He met his first boyfriend when he attended Exeter Art College, but before that he'd had a sexual awakening watching Channel 4's late-night progressive queer programming, Derek Jarman's Latin-language film Sebastiane being one particularly vivid memory: "I wanked a lot watching it," he says.

He didn't really get involved in London's gay scene until later into his 20s, living it vicariously through the queer pop soundtrack of the time; songs such as Bronski Beat's Smalltown Boy, The Power of Love by Frankie Goes to Hollywood and Tainted Love by Soft Cell, whose singer Marc Almond is featured in the show. He eventually started going to Love Muscle at the Fridge in Brixton, "a big gay night of 1,000 people on a Saturday".

Walker found the studios of London's East End to be "crummy" and, after graduating, moved to New York, which was experiencing a boom in fashion photography with the advent of supermodels and high-octane glamour rebelling against 90s minimalism. Here was the home of Irving Penn, Patrick Demarchelier and Richard Avedon, for whom Walker worked.

What did he learn under Avedon's tutelage? "Avedon was very good at communicating with the subject, and he was clear about connection. He would spend a lot of time with the subject before he photographed them." He'd be playful with supermodels, not just telling them "beautiful, sexy!" but giving instructions such as: "OK, you're a crow on a branch, and there's a worm on the floor. I want you two birds to fight over the worm."

Walker has long worked at the very top of fashion and celebrity photography. He's had a longstanding collaboration with Kate Moss and has shot the likes of Lady Gaga, Björk and Frank Ocean for fashion magazine W. He tells me about his most recent W subjects: Rihanna and her baby Rocki. "She's mad!" he says. "She was in Paris looking at the couture shows and said, 'I really want you to photograph me and my baby.'" The shoot was scheduled for 2pm; famously tardy, Rihanna turned up at 11pm. "I've never worked through the night with someone, but it was a vibe," Walker says. "She's someone you could photograph in seven frames and you've got a picture, because she's so bodily aware."

Not all his shoots were so successful: he mentions a 2005 session with Madonna for American Vogue. At the time, Madonna was "obsessed with Cecil Beaton, in her English Lady of the Manor era". They shot at Beaton's old home in Ashcombe, which Madonna had recently bought: Walker photographed her dressed in tweed and cashmere, surrounded by chickens and holding a black lamb "because she's the black sheep of the family".

Walker feels he fumbled the attempt to merge the worlds of Beaton and Madonna. "I look at all these pictures – she's actually not big enough [in the frame]," he says. "But the awful thing is we had three days to do the shoot, and the first day was like a warm-up. The pictures weren't great, but I thought we'll get there. But at the end of day one she was like, 'We've got the pictures, we're done!'"

Madonna later joined Walker and his team in the local pub. When she walked in with then husband Guy Ritchie, "the whole pub fell silent". Madonna then asked Walker what was he was having for lunch. "And she said, 'I'm going to have a pint of Guinness and I'm going to eat something.' So she says 'Excuse me' to the waitress, 'I'm going to get what he's having.' And the young waitress goes, 'I'm sorry, the kitchen's closed.'" Wow, no Madonna privilege? "Not in this pub!"

Walker admits that he's not particularly interested in fashion. His favourite pastimes are reading (Denton Welch's In Youth Is Pleasure and Josh Silver's Fruit Fly are recent favourites) and walking his dog, Jack: "That's my absolute favourite thing to do!" He is far more interested in fantasy and fairytales, lighting up when he talks me through a book of old shoots which draw on Alice in Wonderland and the Wizard of Oz. "Fashion isn't the reason I became a photographer, it was fantasy."

He regrets the fact that glossy magazines are in decline, and that his work perhaps belongs to an era that is dying. On his Instagram, he often reflects on past shoots, but he has a sense of inevitability about the digital world taking over. "We're at the end of that time, which is sad. Looking at photographs in a magazine format is beautiful because of the scale. Just turning a page I lament that passing, but we are where we are."

Fairyland, then, is one way for him to continue working at an ambitious scale. I wonder if, in capturing everyday heroes, he's also revealing an exasperation with celebrity. "I think as a photographer, or any creative, you're always looking for something different," he says. "That pendulum does often swing a long way from where you've been."

He leans over the scale model, closes his eyes, and muses on the six years it took to make: "I've been so nourished by these stories, and I've been moved to tears by these stories." Fairyland may be new work, but for Walker it reframes the span of his career. "Doing this show, I now realise that everything I was doing as a fashion photographer was all camp and gay. It was my gayness escaping."