'Beware … of assuming the sterile attitude of the spectator, for life is not a spectacle" scream the words of Aimé Césaire – writ large and bold by Barbara Kruger, in a giant installation, The History of Tears, that fills Bourse de Commerce's spectacular Rotunda. Kruger's angry, menacing, viscerally charged work – words as images, and large-scale video clips of people seething with self-hate – is a thrilling setup for Remember Me, an exhibition that promises a take on 200 years of photography through 700 works by more than 70 artists drawn from the collection of the French billionaire François Pinault. That excitement quickly dissipates as the exhibition moves on.

This is the biggest photography show ever held at the Bourse de Commerce. Remember Me has two main problems: first, it tries to say too much with too few artists. For a start, nearly 450 of the 700 or so works are by Irving Penn. It leans heavily on classical, conventional approaches, styles and familiar names, and that creates a very obvious skew in the perception of what photography is, does, and how it is presented.

The organisation makes no real sense. There are two main spaces, in addition to Kruger's installation, dedicated exclusively to one artist. Penn is one of them. The installation looks undeniably beautiful: following the curving Ando architecture, the works have been hung like waves, expanding in undulating lines across the walls, without chronology or theme, flowing freely into the space. However, when you look closely at the works themselves, this configuration becomes very problematic. There are a large number of colonial-style portraits taken in a temporary tent studio by Penn when he travelled in Cameroon and Benin. Many of the pictures depict men and women in traditional tribal dress, and were taken as a commission for Vogue. These nations were actively negotiating independence from France at that time. When you compare them with portraits by Malick Sidibé or Seydou Keïta of the same time, they look even more like anthropological, ethnographic, extractive images by a westerner who couldn't even communicate with his subjects.

"Primitive subjects respond more to tone than to obscure words," Penn stated of how he directed the people who appear in these photographs. To me, that says everything about his perspective on African bodies and culture. Then you notice the rest of it: the glamorous white women who appear as pristine models of passive beauty, while all the men are mediocre-looking and wear suits. Still lifes are woven through – presumably to undo the idea of hierarchy between subjects. So, a man who looks at a vase or a plate of fruit the same way he looks at people. The camera has been instrumental in colonialism, and in cultural extraction and appropriation – including in the fashion industry, of which Penn was part. But there's no attempt to tackle that here, to counter these visions with other perspectives.

It's not the last you'll see of Penn, unfortunately – his works keep popping up through the upper floor, where various major names reappear in different sections that each try to heavy-handedly explain and demystify, in an annoyingly didactic way, what photography is and how it's made. We jump between sections on composition, abstraction, portrait and seriality, to nudes – where the women have gorgeous bums and breasts turned alluringly to the camera, and the men are small and, so we're told to think, weak and vulnerable, with very large penises. My biggest issue here was why the sumptuous, elegant nude portrait of Zanele Muholi (by the artist themself, but not technically a self-portrait) needed a lengthy wall text, justifying the work via the western painting tradition – where none of the nudes of white women seemed to warrant one. Yet a quirky series of small 1933 nude shots by Man Ray – an erotic performance featuring the younger artist Méret Oppenheim, unclothed, and Man Ray in a suit and bowler hat – are left with no explanation about the power dynamics of the camera and male surrealism's issues with women.

This is a show that's too afraid to let people think for themselves. We're told Richard Avedon's trio of 1963 portraits of William Casby, a former enslaved person who was 106 when Avedon photographed him, elevate his subject, giving him back his dignity. They're glaringly close and unflattering photographs that resemble mug shots. Compare them with Avedon's smaller, slightly blurred, portrait of Malcolm X. Decide for yourself what Avedon felt, whose image is claimed and whose is handed to them.

Once the exhibition lets go of the binaries, the anxiety to overexplain, or to adhere to the techniques of photography, when it allows active participation or just to be with the work, it gets better. The looser, thematic unions between images are intelligible and interesting, such as sections on "hands" and a glorious room exploring photographs of the sea. These rooms make you feel something, and reflect photographic resonances through history in a more meaningful way. By the Sea, for example, moves seamlessly from the 19th century in Gustave Le Gray's painterly wet collodion prints, where he spliced two negatives together, one for the sea and one for the sky, to create his perfect vision of a seascape, to Hiroshi Sugimoto and Wolfgang Tillmans' elegiac and laconic black and white ruminations. I think about why artists keep returning to look at the sea, looking for the infinite, for continuity, for obliteration and enormity. This room captures the feeling of gazing out at the sea, the ocean's soothing violence and impervious calm.

Photography is an inherently accessible medium, and they take it seriously in France. This show feels fumbling, and an unnecessary dumbing-down. Trying to bend two centuries of one of the most prolific mediums around these particular artists is like trying to tell the history of food through three ham sandwiches. There are more than 8,000 photographs in the wider collection: perhaps it's more varied in perspective than appears in this show – but as it is, it shirks the responsibility of retelling photography's history with nuance and multiplicity. Remember Me forgets Kruger's invocation at the start.

Remember Me is at Bourse de Commerce, Paris, until 18 January