Artists and writers have long been drawn to solitude – but why is that, and what can we learn from them? Beverley D'Silva speaks to Tracey Emin and others to find out more.

From Wordsworth wandering lonely as a cloud to a thousand pop songs about love-lost angst, culture is steeped in references to loneliness. Some authors have sung its praises – Virginia Woolf said loneliness allowed her to feel "the singing of the real world". Emily Brontë's novel, Wuthering Heights, pulsates with agonising loneliness, from its windswept Yorkshire Moors setting to its moody, solitary antihero Heathcliff. The writer is said to have avoided human society, and rarely left Haworth. While comedian Lily Tomlin once quipped: "Just remember. We're all in this alone".

Artists have traditionally spent time apart from others, all the better to connect with their muse. Shakespeare wrote one of his masterpieces, King Lear, in quarantine in the early 1600s, as the Black Plague ravaged London. Frida Kahlo said she painted self-portraits, for which she is most noted, because she was "so often alone". Van Gogh left Paris in the hope that the quieter ambience of Arles in the south of France would give him mental clarity. The lonely, tortured artist archetype can be traced back to Renaissance architect and artist Giorgio Vasari; his influential book of 1550, The Lives of the Artists, depicts the artist as someone who lives on the periphery of society, literally and metaphorically.

Of course, not everyone wants to be isolated, and lockdowns and social distancing rules have inevitably led to greater feelings of loneliness, particularly among young adults. But if loneliness is distressing, then solitude is a more agreeable animal, and individuals have chosen to be alone since ancient times. The first known Christian hermit was Paul of Thebes believed to have lived alone in the desert of Egypt for many decades. Anthony Storr, in his book Solitude: A Return to Self, proposed that spending time happily alone is necessary for mental health and creativity, and that the most profound human experiences have little to do with our relationships with others. And Picasso, no less, is much quoted on the subject, with the phrase: "Without great solitude no serious work can be accomplished".

The loneliness of the soul

Loneliness and grief carve a deep, dark seam through the current show by the British artist Tracey Emin. Her joint exhibition with works by Edvard Munch, at London's Royal Academy, bears the title The Loneliness of the Soul, reflecting qualities she has identified with in the Norwegian Expressionist's work since she "fell in love with him" at the age of 18, she tells BBC Culture.

Munch was very emotional but very existential too, always in a position of being on the outside looking in – Tracey Emin
Emin may still be best known worldwide for her "confessional" and controversial art installations of the 1990s including My Bed, with its cigarette butts, vodka bottles and stained mattress; and Everyone I Have Ever Slept With 1963-1995, appliquéd with 102 names. Not to mention her reputation for hedonism, and stark emotional honesty about her public and private life. As she says: "I tell it like it is". But her art and life – always twin flames – have changed. She's embracing her exceptional talent for painting while recovering from serious illness.

In July last year Emin was diagnosed with squamous-cell bladder cancer, the same disease her mother, Pam, died of in 2016. Emin underwent radical surgery to remove her bladder, uterus, fallopian tubes, ovaries, lymph nodes, urethra and part of her colon and vagina. In April this year she announced on BBC Two's Newsnight that her cancer is "gone". She also described how she faced the rest of her life with a "major disability" (a urostomy bag replaces her bladder). "I never realised how much I wanted to live until I thought I was going to die," she said. In May her "cancer self-portraits" were published in The Guardian, brave selfies taken while she was in hospital and post-surgery.

Prior to her cancer diagnosis, Emin, 57, had been working hard on the Royal Academy show. From thousands of works in the Munch archive in Oslo, Norway, she selected 18 oils and watercolours by the artist. These now sit with more than 25 of her own works, including paintings, sculptures and neons – many made in the same time period, but 100 years apart. It's a landmark show, the first time the two artists have been displayed together.

Its themes of grief and loneliness are palpable, the works full of drama and quite heart-rending, especially in the context of Emin's illness. In the works I am The Last of my Kind (2019) and You Came (2018) there is a visceral vulnerability and anguish in the solitary nude figures; trauma is etched into the powerful This is life without you – You made me Feel Like This (2018); while the earlier neon piece More Solitude (2014) points to the necessity of aloneness.

Her 40-year fascination with Munch parallels their lives and psyches. He painted his emotional state with painful rawness, a shockingly new technique that gave birth to Expressionism. His "soul paintings" – tortured souls and ghostly lovers – reflected his fascination with then-popular spiritualism and the supernatural. That interest offers further common ground – Emin has said her relatives held seances at home in Margate. Munch had a drinking problem and was forced to give up alcohol to save his sanity – Emin has also given up drinking, to save her health.

She sees him as a kindred soul, telling me over the phone from her London home: "I related to how he viewed the world so alone. He was very emotional but very existential too, always in a position of being on the outside looking in." Munch wasn’t very happy, as one would suspect from his work: "He had a lot of relationships but not much sex," she says.

Emin has always been candid about her love life: she tells me she's not been in a relationship for 11 years, and has lived alone for 20 years, then corrects herself: "I lived with my cat, Docket, but he died last year." Docket was her "little soulmate". Now he's gone, she feels there's room, more space to invite love into her life.

Emin has been reading Munch's portrayals of death (he lost his mother when he was five, his sister at 14) in the context of the current pandemic. "He was brought up with a lot of death around him. Think about it, before [the current pandemic] some people had not experienced death unless it was their parents, say. Now few can say that after the past year… People have become much more aware of the finality and the fragility of life too."

That might include herself – and have helped to shape her own experience of solitude over the last year. When she was diagnosed with cancer, she consciously distanced herself from friends to preserve her energy to get through the surgery and to heal she says. During her recovery she's been working less and has allowed friends back in – she has been glad of their support. "I'm the happiest I've been, really." Facing death has "made other problems pale into insignificance. If something made me unhappy before, it would engulf me. Now I just deal with it… The cancer's really helped me with that."

The artist is relishing a new lust for life. Her 9m-high bronze sculpture, The Mother – "I'm giving Munch the mother he never had" – will be installed outside the Munch Museum in Oslo; her studio complex in Margate should soon be completed; and a major show in Venice, which was postponed owing to her illness, will now go ahead in 2024. Also, she says: "I'm open to love, which is very different to how I felt… I also feel I deserve it, maybe before I didn't."

Her psyche has changed; she feels that her ordeal has erased trauma. "It's gone," she says. Emin appreciates her "amazing life….I'm embracing everything. Nearly dying has a lot to do with it, I know." She's working on having a "quieter, more peaceful life", and plans to fill her days with "painting to the sound of birdsong". She's sure of one thing: "I no longer feel alone… I think the chance of me dying without someone holding my hand is zero."

Tonic of wilderness

According to many artists and writers, there is a "cure" for loneliness. Maya Angelou said music was hers: "I could crawl into the space between the notes and curl my back to loneliness". Others find it in nature, hiking, wild swimming: the "tonic of wilderness" as Thoreau put it.

There's so much stimulation in the wild, maybe there's no space for loneliness – Martin Shaw
Author Martin Shaw knows about the virtues of isolation outdoors. As a "rites-of-passage wilderness guide", he takes groups into deepest Dartmoor, in the English countryside, and other remote places, where they separate and learn to "reconnect with nature and their own character". Retreats like these date back to Ancient Celtic times, says Shaw, the author of Smoke Hole: Looking to the Wild in the Time of the Spyglass, which reframes myths for our times. He has also collaborated with the actor Mark Rylance and the artist Ai WeiWei.

Shaw tells BBC Culture that people on his retreats, from troubled teens to traumatised war veterans, can report being bored or homesick but "I've never had anyone come back and say they were lonely… There's so much stimulation in the wild, maybe there's no space for loneliness." Shaw is at home outdoors – he lived in a tent for four years – but despite his deep inner resources, found the confines of his Dartmoor home in the third lockdown a bit much. "I genuinely was lonely and missed conversation and simple interactions."

The key is in forging meaningful relationships, according to Dr Andrea Wigfield of the Centre for Loneliness Studies at University of Sheffield. The centre's research showed that when people can't be together, arts-based activities, such as children and grandparents working together online on arts projects, can help them to connect. However these aren't enough on their own to combat loneliness if meaningful relationships are absent. In the UK, Loneliness Awareness Week (LAW) 2021 runs from 14 to 18 June. Run by the Marmalade Trust, the theme this year is acceptance, and their "We get lonely" campaign aims to make loneliness accepted as a natural part of being human.

The writer Olivia Laing is familiar with finding positives in adversity. Moving to New York City in her 30s after a break-up, she felt a terrible loneliness every day. The book that evolved from that, The Lonely City: Adventures in the Art of Being Alone, was a hit. The New York Times said it was "daring…about the universal struggle to be known". In it, Laing examines loneliness through art, with essays on Andy Warhol's Time Capsules and Edward Hopper's Nighthawks among others. Laing tells me that focusing on Hopper's paintings with their "feeling of separation, of being walled off or penned in, combined with near-unbearable exposure" helped her, and "eased the burden of my own feelings. Someone else had grappled with loneliness, and found beauty, even value in it".

More about: