Encryption is a vital part of our digital world these days, to the point that it’s impossible to imagine online life without it.
Many of us use it when we’re storing or sending data, for example via BitLocker on our hard disks and HTTPS in our browsers.
Has this given us a false sense of security, by luring us into assuming that if our data is safe a lot of the time, we can act as though it is safe all of the time?
The art of secret writing
Encryption, or the art of secret writing, has been around quite literally for millennia.
Historical evidence suggests that Egyptian tomb-carvers had tricks for disguising their written records almost 4000 years ago, that the Mesopotamians used a form of cryptography to protect trade secrets back in 1500BC, and that the military in ancient Sparta had a system for scrambling messages to keep them secure when they were carrried by runners travelling through hostile territory.
Julius Caesar, who was assassinated in 44BC, was very keen on encryption, and used an algorithm that bears his name to this day, the Caesar cipher.
We know about the Caesar cipher because it was publicly documented within 200 years of his death by the writer Suetonius, who provided an algorithmic hint on how to get started on cracking any surviving secret material from Caesar’s lifetime:
If there were matters he wanted to keep secret, he wrote them encrypted by re-ordering the letters so that no words could be made out.
But right after this, Suetonius went one step further and revealed the specific letter-shifting distance that Caesar himself had chosen for his encrypted texts, thus retrospectively giving away the decryption key for all Caesar’s correspondence:
Anyone wanting to investigate and decrypt this cipher should shift the letters four places along, thus writing D in place of A, and so on.
In modern English, we usually describe the Caesar cipher as having a shift distance of three, given that D is three places along from A in the alphabet by our reckoning. But the Ancient Romans counted inclusively, tallying the total number of items in a sequence, not merely the number of steps to get from the first to the last. Instead of starting from A and going >B>C>D = 3 places, they started with A and counted A>B>C>D = 4 places.
Even if Caesar’s letters had remained secure until Suetonius’s book De Vita Caesarum (Lives of the Caesars) came out in AD121, that single sentence blew the lid off Caesar’s secrecy forever, giving everyone an instant answer to the riddle, including readers with no cipher-cracking skills of their own.
Today, this retrospective bulk cracking of historical data has become a serious cybersecurity concern, and has led to a number of cryptographic techniques known in the jargon as forward secrecy, aimed at making the raw data from an encrypted conversation useless once the conversation has finished.
In other words, if you weren’t party to the conversation and couldn’t successfully decrypt it at the time it was transmitted, you shouldn’t be able to recover any left-over decryption secrets to help you unscramble it in the future, even if you managed to record all the encrypted data in the hope of cracking it later.
By the way, some writers still use the more dramatic-sounding term perfect forward secrecy, but we urge you to avoid that phrase, given that very little in cybersecurity is truly perfect.
The software and hardware used to implement ‘perfect’ cryptographic algorithms may be imperfect; the algorithms themselves may turn out to have unexpected security holes; and cracking tools that might have needed uncountable millennia to run 50 years ago might now take only a few hours on the sort of laptop you can buy for $1000.
The trouble with tools
As you can probably imagine, encryption techniques have left cryptographers on the horns of a dilemma right from those early times.
Like many technological advances, from improved military weapons such as breech-loading cannons and select-fire rifles, to better industrial tools such as more powerful steam engines and more efficient electrical generators…
…they represent a huge advantage when you have them, but you lose that advantage if they get hold of them too, whether they means your military enemies or your commercial competitors.
Strong encryption certainly helps you keep your secrets safe from snooping and interception, which can give you a significant advantage, both in war and in peace.
But surely that means you need to keep secret the process for keeping your secrets, too?
How can you do that once you have deployed your system, as Caesar did when sending letters to his confidantes throughout Italy and the Roman empire, or as the Nazis did with the infamous Enigma cipher machine during World War 2?
Clearly, every sender and every recipient needs to know how the system works, and may need a team of clerical staff to operate any necessary equipment, which makes it as good as impossible to keep the system itself secret.
The German military, for example, is estimated to have fielded about 40,000 Enigma machines during the late 1930s and early 1940s.
Until fairly recently, the knowledge of the inner workings of an encryption system often gave attackers a huge helping hand in developing ways to crack the process without knowing the encryption keys in advance, either by working backwards from the scrambled text to recover the keys that were used, or by sidestepping the keys altogether by finding a shortcut to do the unscrambling without them.
Cracking Caesar
As an example of how merely knowing the details of an insecure algorithm may help you crack it more quickly that you first thought, let’s revisit Caesar’s cipher.
We now know, thanks to the big reveal by Suetonius in the second century AD, that Caesar was guilty of something that we are all aware we shouldn’t do (yet some of us do anyway, to save time), namely reusing passwords.
Caesar stuck to a letter shift of three, or four if you prefer to follow the nomenclature of the time, for all his correspondence.
He didn’t agree a different shift distance with each recipient, so even if all the recipients had kept the shared key to themselves, they would nevertheless have been able to encrypt everyone else’s private letters to and from Caesar.
But using a different key for everyone doesn’t help very much once you realise that letters are only ever shifted by exactly the same distance each, rather than shuffled using a random permutation.
If you shuffle the alphabet randomly (we’ll use today’s 26-letter English alphabet, including the letters J, U and W that Caesar didn’t have), you have 26 choices for the letter to represent A, then 25 choices for the letter to replace B, then 24, and so on down to a solitary ‘choice’ left at the end for the last remaining letter that will stand for Z.
So you have a whopping 26×25×24x···×3×2×1 different ways to do the scrambling, which comes out at a number with 27 decimal digits, or 89 bits.
That’s too many to try them all, even on a modern computer.
But with a Caesar-style shift, you only get 25 different shift patterns before you are back to the unencrypted alphabet:
That’s because shifting by 26 is the same as not shifting at all; shifting by 27 is the same as a ‘null shift’ of 26 followed by a shift of 1; and so on.
Once you know that there are only 25 different possibilities – in modern jargon, that the keyspace is only 25 – then it’s a trivial matter to try each of them in turn until the right answer comes out.
Knowing that the Caesar cipher was used rather than a randomized substitution alphabet means that cracking the message will only ever take a matter of moments, and even in the worst case will take you just 25 times longer than if you knew the key in advance.
Don’t hide your code
Based on the discussion above, it sounds as though keeping your encryption algorithm secret should help you in two ways: firstly by making the encryption system tougher to probe for weaknesses and harder to crack, and secondly by preventing your enemies from using your own technology against you.
As we mentioned above, however, it’s as good as impossible to keep your algorithm away from prying eyes once you have distributed it to anyone else with whom you plan on communicating, especially if it’s implemented in the form of software that can be used on a wide variety of different operating systems.
On the other hand, you genuinely do have a fighting chance of keeping your encryption keys secret, not least because you can change them as often as you like, in contrast to the cryptographic hardware or software you use, which can’t be modified at will.
Relying on analysts and attackers having no access to your source code or to your devices is an example of what’s known as security through obscurity, which is a bit like relying on house burglars not thinking of looking under your front doormat to see if you left a key there for emergencies.
As long as no one looks, you can pretend that the emergency key is safe, but as soon as one person decides to take a peek, the secret is out of the bag for everyone, and your illusion of security vanishes at once.
The inadequacy of cryptography based on a foundation of obscurity was formally documented back in 1883 by a French-based Dutch academic by the name of Auguste Kerckhoffs, and his warning that the cryptographic system itself must not be part of the secret is today known as Kerckhoff’s Principle.
This principle is a fundamental rule for modern encryption systems:
As an aside, if anyone tries to sell you an encryption system that relies on a secret algorithm, or a proprietary one that requires you to sign a non-disclosure agreement before you are allowed to find out how it works, don’t buy it. (In fact, run away from it, don’t walk!)
Almost anyone can devise a cryptosystem that they themselves aren’t able to crack, but the fact that they can’t break it doesn’t magically mean that no one else can.
Remember that, in real life, you won’t be protecting your data from that one vendor, but from anyone and everyone else.
There are many publicly available, patent-free, open-source, widely scrutinized, and officially trusted algorithms available, so simply go for one of those instead of putting your faith in someone else’s secretive and unverifiable claims.
Cryptography as good as you need
In 2024, we can all readily pick from a surprisingly wide range of cryptographic algorithms that are entirely public.
All those algorithms are available in numerous, independently created, open-source software versions, and were developed in the glare of intense practical and mathematical scrutiny by recognized cryptographic experts from a wide range of different backgrounds, cultures and countries.
It’s improbable, to the point that you can consider it impossible, that all of these experts could have colluded to reach a dishonest artificial consensus, assuming that any of them wanted to.
In simple terms, if we don’t try to knit our own encryption, choose wisely from the respected algorithms already available, and use those algorithms correctly in our own software…
…we should end up with strong encryption that is computationally infeasible to crack, as the jargon puts it.
The words ‘computationally infeasible’ don’t tell us that an encryption algorithm is truly unbreakable, merely that it can be used with the same confidence as if it were.
As Kerckhoffs stated nearly 150 years ago in the very first point in his list above, “The system should be practically, if not mathematically, indecipherable.”
In theory, therefore, contemporary cryptography can be considered a cure for our collective concerns about the secrecy, privacy and integrity of our personal information, both in transit and at rest.
However, if we can do it, then they can do it too, where ‘they’ covers anyone and everyone with our worst interests at heart.
That list includes cybercriminals, stalkers, spammers, scammers, sex offenders, money launderers, unscrupulous companies, oppressive governments, corrupt officials, terrorists, and more.
In practice, therefore, some people consider contemporary cryptography to be a curse that aids and abets criminality, emboldens criminals of the most odious sort, and hinders the investigation and prosecution of evildoers.
The dilemma is back
The cryptographic dilemma is back in a new form.
Strong encryption can make all of us acceptably secure online, so many governments and regulators are working to ensure that we all make use of cryptography for exactly that reason, including imposing legal and financial penalties on companies and individuals who don’t take security seriously.
But strong encryption, some insist, makes criminals and internet ne’er-do-wells too secure online.
At the same time, therefore, many governments and regulators are arguing vocally that we need weaker encryption that can be cracked if necessary, and threatening legal and financial sanctions against individuals and companies who don’t want to play the game that way.
Ironically, in some countries the public sector almost seems to be at odds with itself, with some government departments calling tirelessly for exactly the opposite technical outcome that other departments are strenuously insisting we need.
And there’s a third part to the dilemma, if that’s not a contradiction in terms (strictly speaking, the di- in dilemma means exactly two, not two or more).
Those of us who are business owners have free access to as-good-as-undefeatable cryptography if we take the trouble to implement it correctly.
And those of us who are computer users already enjoy the benefits of strong encryption in many parts of our online lives, such as HTTPS to protect us from snooping while we browse, and tools such as BitLocker (on Windows) and FileVault (on macOS) to keep our files secret if someone steals our laptop.
Nevertheless, as we investigated last week, we seem to be experiencing ever-bigger data breaches ever more frequently.
These breaches almost always involve the attackers getting hold of unencrypted copies of critical personally identifiable information (PII).
This PII often includes data such as social security numbers, home addresses, scans of ID documents, messaging histories, photo albums, email archives, and sometimes even our passwords, which aren’t ever supposed to be stored by anyone unless they’re strongly protected by cryptographic means.
That’s another aspect of the curse-versus-cure problem with encryption.
When we see strong encryption all around us in our digital lives, should we feel upbeat because we’re improving our resilience to cybercrime?
Or should we feel worried because we’re experiencing what some privacy experts refer to as ‘security theater’, leaving us with a false sense of protection because of imperfect and incomplete implementations?
What to do?
Loosely speaking, some encryption is better than none, provided that we are aware of the limitations of what we have, and don’t get lured into a false sense of security.
For example, credit card fraud is harder in a world with Chip-and-PIN, where a secure chip on your credit card adds a layer of cryptographic protection to each transaction, than it was when we relied on credit card magstripes, which could be read out with ease, in plaintext, by anyone with a $10 card reader.
But chipped payment cards aren’t enough to stamp out fraud on their own, because there are still dozens of other ways for crooks to acquire your card details without reading them from the card itself, such as phishing, malware that logs keystrokes, and data breach attacks against payment services.
As another example, internet surveillance and snooping is much more difficult when you use a modern browser to visit sites using HTTPS than it was when anyone with access to almost any router or switch along your network path could trivially record every byte of all your online activity in plaintext form.
But HTTPS is only a partial precaution, because it protects your privacy between your browser and the server, but doesn’t stop criminals from setting their own rogue HTTPS sites to serve up malware, or from using HTTPS themselves in their malware so you can’t easily spot when they’re uploading stolen data from your network.
So, there are two important things to remember:
- Encryption isn’t a panacea. There are some security outcomes that you cannot achieve without it, such as shielding your personal information from trivial interception when you shop online. But there are other security outcomes that encryption alone cannot provide, such as preventing successful cyberintruders from stealing your trophy data. It’s good news if you keep your customers’ data strongly encrypted on your database servers, but bad news if there’s a backdoor into your network that allows attackers to login illegally and query that data in its decrypted form.
- You can’t strengthen things by weakening them. Government proposals aimed at legislating deliberate weaknesses or backdoors into our cryptographic systems aren’t new, and are unlikely ever to go away. Sometimes, these are well-meant, and their proponents often have honorable intentions such as catching stalkers and child abusers, or uncovering terrorist plots and plans. But the problem with these proposals is that they violate Kerckhoff’s Principle by making the system itself part of the secret, instead of relying for its security on cryptographic keys that can be changed whenever necessary.
Taking a stance against government-mandated cryptographic backdoors may feel like a somewhat anarchic or countercultural position, and those who go down that path sometimes find themselves derided as ‘traitors’, ‘scofflaws’ or ‘aiders and abetters of crime.’
But backdoors, even under government control, are almost certain to make things worse for us, because they will figure out those backdoors sooner or later, while at the same time ignoring the regulations (they’re criminals, don’t forget!) and using unbackdoored cryptography themselves.
Interesting times!
Why not ask how SolCyber can help you do cybersecurity in the most human-friendly way? Don’t get stuck behind an ever-expanding convoy of security tools that leave you at the whim of policies and procedures that are dictated by the tools, even though they don’t suit your IT team, your colleagues, or your customers!
More About Duck
Paul Ducklin is a respected expert with more than 30 years of experience as a programmer, reverser, researcher and educator in the cybersecurity industry. Duck, as he is known, is also a globally respected writer, presenter and podcaster with an unmatched knack for explaining even the most complex technical issues in plain English. Read, learn, enjoy!
Featured image by Marcos Mayer via Unsplash.
