'Criminal trials without juries are a bad idea… Dispensing with juries will damage our democracy… You don't fix the backlog with trials that are widely perceived as unfair.'

Not my words, but the words of David Lammy. Yes, the very man who announced yesterday he is slashing the number of jury trials in half.

It's quite the about-turn. And it's not just the Justice Secretary. Writing on behalf of (who else?) the Society of Socialist Lawyers in 1992, a young Keir Starmer urged for there to be a 'right of trial by jury in all criminal cases'.

The future PM acknowledged that expanding access to jury trials would result in an 'inevitable increase in costs' – but argued that the right was so fundamental, we should proceed anyway. How times have changed.

As is so often the case with this Labour Government, it says one thing and does another. Following Calamity Lammy's announcement, juries will be used only in cases where a defendant is likely to receive a sentence of more than three years. At least on this occasion Labour can't be accused of lying, because nobody thought to ask them if they would be so reckless as to tear up our ancient liberties.

There's a good reason that jury trials have endured for so long – since before Magna Carta in 1215. They pool the collective wisdom of 12 ordinary citizens and thereby give a voice to the common-sense sentiment of the British people. For centuries, they have served as a safety valve against an overbearing state. As the jurist Lord Devlin once said, they are 'the lamp that shows that freedom lives'.

In February, Jamie Michael, a former Royal Marine, was cleared of 'hate speech' charges by a jury in Merthyr Tydfil. It took them just 17 minutes to find that his offensive Facebook post shouldn't land him in prison.

Would he have received the same common-sense hearing from a judge? It's impossible to say for sure. But when we look at cases such as Lucy Connolly's – the wife of a Tory councillor who spent nine months in prison for an offensive tweet during last summer's riots – it's clear that the British people have a clearer, instinctive understanding of justice, including on issues such as free speech, than much of the legal establishment.

The Free Speech Union pressure group has pointed out that if people charged with speech offences are denied this right, they're more likely to be convicted. According to their research, those using free speech as a defence are more than twice as likely to be acquitted in crown courts, which often use juries, than in magistrates' courts.

So Lammy's bizarre idea isn't just an assault on an ancient British right – it represents a doubling-down on our absurd speech laws, with judges alone empowered to decide whether you should go to prison for an offensive post online.

The excuse that Lammy is invoking for his changes is the spiralling court backlog. It is up 10 per cent in the past year and some cases are listed as far away as 2030. The delays are ruining the lives of victims and making convictions less likely. It's undoubtedly true that radical action needs to be taken.

But his excuse simply doesn't stand up to scrutiny. The Criminal Bar Association, among many others, has pointed out that jury trials are not the reason for the backlog. There is no conclusive evidence that scrapping them would make even a dent.

When I pressed him in the House of Commons yesterday, Lammy couldn't even say his changes would lead to a reduction in the backlog by the next election. This is clearly a distraction, attacking one symptom while leaving the disease untouched.

If Lammy was serious about fixing the crisis, he would get our courts sitting around the clock. Yesterday, more than 50 crown courtrooms across England and Wales sat empty. This year alone, more than 21,000 court days have gone 'unused'. If the Government pulled its finger out and filled these courtrooms, it could have brought down the backlog by up to 10,000 cases this year alone.

Lammy could reform the probation service, overhaul court listings and tackle the appalling delays that result from late prison transfers. As with so much of the British state, inefficiency is endemic.

Be in no doubt: this is a choice. At the Budget, Rachel Reeves found billions more for benefits, but not the sums needed to invest in our justice system. Last year, the entire budget for courts and legal aid was £5.5billion. That's almost exactly the same amount – £5.4billion – that we spent looking after illegal migrants who've broken into our country. When push comes to shove, this Labour Government is suspicious of anything that allows the British people to have their say. After all, it was Keir Starmer who spearheaded the campaign for a second EU referendum. This is just the latest example of Starmer's arrogant lawyers-know-best elitism.

We see it in the surrender of the Chagos Islands, the result of an advisory judgment from the International Court of Justice. We see it in the refusal to leave the European Convention on Human Rights, despite the fact that our continued membership makes it impossible to stop the boats and deport countless foreign criminals. It's absurd that Lammy will defend our continued membership of the ECHR, but won't support your centuries-old right to a trial by jury.

Rights derived from foreign courts and treaties are seemingly sacrosanct, while home-grown ones can be carelessly cast aside. Instead of putting our faith in lawyers, we should trust in rights and principles that have endured for centuries, and in the British people themselves.

The truth is that Lammy and Starmer are curbing jury trials because they want to. They don't trust ordinary people.

But our ancestors didn't fight for our rights, only to have them taken away, centuries later, by this Labour Government. We must resist.

• Robert Jenrick is Shadow Justice Secretary