This year, European leaders marked 60 years since the Treaty of Rome, one of the foundations of the European Union (EU), by both acknowledging the EU's achievements and admitting the many challenges it faces.

Throughout its existence, the Union has weathered crises, but the current culmination of challenges, which has at times led to predictions of regional disintegration, has never been seen before. How will recent events shape the future of the EU?

Facing high unemployment, especially youth unemployment, in many of its member states, large amounts of public debt, nationalist and populist parties gaining ground and continued refugee inflows, the EU's leaders and citizens have to contend with a wide range of crises. Russia is increasingly seen as a direct threat, particularly in eastern Europe, and although economic growth is gradually returning, growth rates are still low. Italian and French markets are shaky and Greece is still close to default.

The leadership change in the US and President Trump's positions on the EU, NATO, Russia and global trade have been unpredictable at best and damaging at worst, suggesting a revised transatlantic relationship in the future. President Trump's preference for bilateralism may also have serious consequences for European unity. German Chancellor Angela Merkel has stated that Europe must take its fate into its own hands and cannot fully rely on the US – a position shared by France's new President Macron. This is a substantial break with the past, especially the close German relationship with the US.

And then there is Brexit. The British referendum on EU membership illustrated deep elite and public scepticism about European integration and its benefits. But the UK has always had a unique relationship with the EU, one that can be described as distant, awkward and conditional. So although some other politicians in Europe had declared they would like to follow the UK's lead, many member states have a different historical relationship with the EU. In the wake of the Brexit vote, support for the EU actually spiked in almost all 27 member states.

Yet there is little cause for complacency in Brussels and throughout Europe. The European Commission's White Paper on the future of the EU, released in March, acknowledged that "many Europeans consider the Union as either too distant or too interfering in their day-to-day lives. Others question its added-value and ask how Europe improves their standard of living.".

This is a remarkably frank assessment by the EU's bureaucracy in Brussels. Although the White Paper cites promising poll numbers of 80 per cent citizen support for the EU's four freedoms associated with the single market, and 70 per cent for the Euro in its area, citizen trust in the EU has decreased – approximately one-third of citizens trust the EU today, compared to approximately 50 per cent 10 years ago.

So has the EU lost direction? Is it experiencing a retreat to nationalism and self-interest? Beyond 2017 will the EU run the risk of collapsing or will it be able to regain the trust it has lost and find its sense of direction again?

The 60th anniversary of the EU coincides with a stream of national elections in Europe, most of which involve trends towards right-wing, populist, nationalist and anti-EU candidates and parties.

In the first round of French presidential elections, Le Pen's National Front achieved the party's best-ever result. As a Franco-German alliance has historically been at the centre of the EU, it was feared that a win for Le Pen might damage traditional conceptions of European integration – a prospect never before even anticipated.

Germany is holding parliamentary elections in September, and while there is little chance that the AfD (Alternative for Germany) will pose a major threat to the country's leadership, it is expected to win some seats in the parliament for the first time and its position is gradually heading further right. The predicted landslide result for Theresa May in the general election on 8 June never eventuated in the UK. Following Matteo Renzi's resignation in December 2016 after being defeated in a referendum proposing constitutional reforms, Italy must also hold elections before 2018, with two parties supporting anti-EU positions – the Five Star Movement and the Northern League.

Yet there is reason for some cautious optimism. Macron defeated Le Pen, although she won some 11 million votes in the second round. The Austrian presidential election in December 2016 saw left-wing politician Alexander Van der Bellen defeat populist Norbert Hofer, whom many observers had thought would have been successful in the wake of Brexit and the election of Donald Trump. And in the Dutch election in March 2017, Geert Wilders' anti-European Freedom Party won fewer seats than expected.

There is a clear voter dealignment from both established party systems and traditional parties. Voting trends point towards gains for populist parties, even if they do not manage to ultimately win government office. For the future of the EU, it will be crucial to actively tackle the causes of a populist turn in Europe before the next round of elections. The populist parties and movements – of both the right and the left – are espousing narratives of a society that is deprived of benefits, with the EU even seen as responsible for losses of sovereignty and control.

The elections in Europe thus far, rather than a left-right contest, have largely reflected a globalist v nationalist, Europe v 'us' contest of nationalism, identity and control. In the first round of the French presidential election, for example, the centre-right Republicans and the centre-left Socialists – both of which had dominated French politics since the 1970s, only garnered 26 per cent of the vote, in contrast to the 2012 elections where they had obtained almost 56 per cent.

Declarations of greater European unity and solidarity will only be possible with a concerted effort to substantially revise the relationship between the nation and the EU, and the EU and the citizen. Emotional reactions to the EU are now just as important, and if not more so, than elite visions of solidarity and transnational cooperation.

For some time integration could be equated with economic prosperity, but when this slowed, regional and national leaders failed to emphasise the added value of the EU in terms of solidarity and security, and instead a narrative of perpetual crisis took hold. The EU is no longer regarded as a solution to political and economic problems, but as their cause.

A new narrative for the EU, and a concerted public diplomacy drive that clearly and honestly allows for dialogue and debate on the necessity, benefits, expectations, added-value, failings and accomplishments of the EU is essential to begin regaining the trust of its citizens.

At the national level, a new dynamic needs to be fostered between domestic political competition and cooperation at the EU level. David O'Sullivan has suggested that there has been a "tendency in all national politicians towards … nationalisation of success and the Europeanisation of failure".

Scapegoating the EU for political gain is a popular game played by many national leaders. This is an ongoing challenge not only for the EU but for all regional organisations and multi-level governance contexts, and is one which can only be overcome by political will, transparency and public trust.

This article has been co-published with the University of Melbourne's Election Watch.

The Rethinking governance in an era of global insecurities, regional tensions and rising nationalism conference is running from 17 - 19 July 2017.

Banner image: Theophilos Papadopoulos/Flickr