HTTPS, short for secure HTTP, shields your browsing from both snooping and tampering, so it sounds like something that every cybersecurity-conscious company would have jumped to provide for its website visitors as soon as it possibly could.
But in an early example of a ‘Stop The Insanity’ story, HTTPS needed at least two decades to take hold, for a bunch of curious and sometimes contradictory reasons.
Join Paul Ducklin for the conclusion of this peculiar but educational tale…
In Part 1, we dug into the history of web browsing security.
In the early days of the web, from 1990 to 1995, browsing was always unencrypted.
Browsers used a refreshingly simple, text-based system known as HTTP, short for hypertext transfer protocol, to request web pages, themselves written in an easy-to-produce content description language known as HTML, short for hypertext markup language.
The idea was to make it easy for anyone to become a website creator, without needing to learn complex typesetting or page layout software.
This wasn’t just text with links, it was hypertext with hyperlinks!
You didn’t need to (and, in fact, in the early days, you couldn’t) specify which fonts to use, or how to arrange and format paragraphs, or how to jazz up the look of your clickable links to make them more attractive than the links on your competitors’ sites.
You simply marked up regular ASCII text files, which you could create in any basic text editor, to tell the browser which parts were headings, where paragraphs should end, which text could be clicked on to hyperlink to another website, and so on.
The hard work of laying out, typesetting and displaying the page, known in the jargon as rendering it, was left to the browser, thus making web pages easy to build.
You could used software you already had, from NOTEPAD and the humble EDIT programs on Windows and DOS, to the ubiquitous vi and emacs editors on Unix systems.
This approach also made web pages much more portable between different computer systems.
Instead of saying, “I want this text to be set in this specific italic typeface, which I hope is installed on your device,” web page authors just said, “Emphasise this text in some way.”
Browsers on computers with graphics displays and high-quality bitmapped text might choose to use an italic font.
Browsers on text-mode terminals might only have a choice of inverse text or basic colors.
Mainframe consoles might be limited to WRITING IN CAPITALS or *adding asterisks* around the emphasis.
Loading HTML files from your own hard disk was easy, because browsers could simply access the required files by name and read them in directly, using traditional operating system calls such as open() and read().
But relying on file-based access to load hyperlinked web pages from other people’s web servers would never have worked out, because few organisations ran network-based file and directory services such as NFS, AFS and SMB (the Windows file sharing protocol now known as CIFS) over the internet.
Problems with performance and bandwidth made the file sharing services of the early 1990s unsuitable for internet use, even if you ignored the risk of rogue outsiders accessing servers and data they shouldn’t.
By defining a very simple HTTP protocol, the fledgling web made it easy for researchers to share pages on their websites without needing to set up network file sharing services first.
(HTTP servers weren’t, and still aren’t, immune to cybersecurity vulnerabilities that could give attackers a foothold, but a basic HTTP server is much easier to code, and in theory much easier to secure, than a general-purpose network filing system.)
File systems typically need to deal with logins, access control, authentication, directory traversal and filename queries, along with giving users the ability to open, close, create, read and write arbitrary files, including allowing the same file to be opened remotely by more than one user at the same time without corrupting it.
In contrast, the original HTTP protocol, as specified by the web’s inventor Sir Tim Berners-Lee, was a model of text-based simplicity:
GET, followed by the path of the URL you want, followed by an end-of-line marker. Wait for a reply to arrive and read in everything until the other end closes the connection. Give the server 15 seconds of leeway to start sending data. Render and display the reply.Importantly, Sir Tim ended his specification with this sentence:
Requests are idempotent. The server need not store any information about the request after disconnection.
The glorious word idempotent is borrowed from mathematics, and loosely denotes an operation that leaves things the same after it’s finished.
For example, the integers zero and one are idempotent under multiplication, because 0 × 0 = 0 and 1 × 1 = 1, but all other numbers give a different value when multiplied by themselves.
In computer science, the word is generally used to refer to a request that produces the same result no matter how often you repeat it, regardless of what you do in between.
In other words, every time you asked the server to GET /content.html, you’d get exactly the same page back, no matter which other pages you’d visited in the meantime.
There were no rotating ads, no content changes based on the last page you visited, and no tweaks based on tracking your actions.
Given that web pages accessible via HTTP were meant to be public, and given that browsing traffic didn’t include any special headers or additional data about your identity or your browsing history, encryption wasn’t considered important back then.
Snoops could monitor what you were interested in, and acquire a list of what you looked at and when, which wasn’t ideal.
But with no logins or passwords unique to your account, and no cookies tracking your previous activity, the danger posed by surveillance was low.
In Part 1, we showed a Wireshark network capture of a modern-day HTTP/1.1 request, showing the typical structure and headers of a visit with no cookies or other personal information included:
An original-style HTTP interaction, now referred to as HTTP/0.9 although it had no designated version identifier at the time, was simpler still, with a single line of text in the request and an unadorned reply including only the HTML of the requested page:
The web proved popular and useful in the scientific and academic community (Sir Tim was based at the nuclear research group CERN in Geneva at the time), but it really took off once money-making companies recognised its commercial potential.
As often happens, this meant features and tweakability in place of minimalism and consistency.
The noble simplicity and repeatability of HTTP’s idempotence made “features” such user accounts, passwords, visitor tracking, targeted ads and online shopping baskets impossible because of its so-called stateless protocol.
Every visit stood alone and played out identically, uninfluenced by anything you might have done, or any interest you might have shown, in the past.
It didn’t take long for this statelessness to vanish: developers at the long-defunct browser Netscape, which was a multi-billion dollar business at the time, hastily added various web browser features that heralded the web as we know it today.
HTTP headers, notably cookies that your browser quietly inserted as additional data in every web request, meant that servers could track what you had done in the past, as well as allowing a site to “remember” whether you had previously logged in.
Suddenly, web requests and replies were carrying personal data about you, not merely copies of publicly accessible research papers you happened to be interested in.
Web-based accounts meant a need for usernames and passwords, and online commerce meant capturing personal information such as delivery addresses and payment card details.
For example, the HTML served up in this HTTP connection creates a form that can you can fill in and upload by clicking on the submit button, which is labelled with the word Pay:
The page shows up in your browser like this:
Because HTTP itself is a largely text-based protocol designed to shuttle raw web requests and replies back and forth, it doesn’t include its own encryption layer, so any data entered into the form…
…gets transmitted in unencrypted form.
Although it’s lightly encoded to stop special characters such as spaces inside a URL from being corrupted or misinterpreted, the original data is transmitted in what’s known as plaintext form, so it can be sniffed out by attackers along the way:
Above, you will also notice a header known as Referer, which the original HTTP protocol didn’t and couldn’t provide, which by default supplies every web page you visit with a record of the page from which you clicked through.
The mis-spelling of the header name (it should be Referrer, of course) is the result of a typo that entered widespread use before the specification was formally published in 1996. By then, it was considered easier to live with the mistake forever than to change it retrospectively.
So that they could send passwords and payment card data over the open internet without drawing the ire of credit card companies, Netscape’s programmers came up with SSL, short for secure sockets layer.
This provided an optional layer of encryption, now known as transport layer security, or TLS, inside which HTTP requests could be “wrapped” if they included any personal or secret data.
You could still type URLs into your browser in the form http://example.com/, and unencrypted network requests would be made to the specified server (by default, these requests use TCP network port 80).
But by changing the URL to start with https://, you could instruct your browser to connect via TCP port 443 instead.
If the server answered, signaling that it was willing to accept encrypted requests, your browser and the server would then negotiate encryption keys for a secure data connection in what’s known colloquially as a cryptographic handshake.
Once they had agreed on the settings needed for an encrypted channel, known in the jargon as a tunnel, they could conduct exactly the same sort of HTTP conversation as they would have done over port 80, but in a shrouded form.
Submitting a login page like this one, where the form upload explicitly specifies an HTTPS connection, rather than plain old HTTP…
…generates a plain old HTTP request inside the browser, but the HTTP data gets wrapped in the security layer provided by HTTPS just before it exits the browser.
This encryption wrapper is only peeled off after the traffic has reached the server, thus providing what’s known as end-to-end encryption.
Anyone between the browser and the server sees the digital equivalent of shredded cabbage instead of the username and password data that was entered:
As we explained last time, however, converting an entire fleet of web servers and services to HTTPS was simple in theory, but unpopular in practice.
HTTPS added unwanted costs.
Server management became more troublesome, because you had to distribute cryptographic keys securely to every computer that might ever serve up a page from your web properties.
And performance dropped, because the S in HTTPS required extra processing time to encrypt and decrypt the data, and extra network bandwidth to set up each encrypted tunnel in the first place.
A partial approach was therefore adopted by many online services, notably including popular social networking sites, even though you might have expected companies with that much influence and processing power to set high standards by preaching the importance of HTTPS for everything.
The trick was to use HTTPS pages when truly private data such as passwords were being exchanged, which looked good to penetration testers and sysadmins, but to revert to HTTP for everything else.
But part-time encryption is never the right approach, and with hindsight the internet giants of the day should never have allowed themselves to pitch it to us as if if were.
After all, the text of every webmail you send and receive, or the content of social media postings intended for closed groups of friends only, is none of anyone else’s business and deserves to be encrypted, too.
There’s not much point in an online messaging service protecting your password at the start of a session if it goes on to expose everything you say thereafter anyway.
Even worse is that once they’ve asked you for your password, websites need to tell your browser to include some magic code, or authentication token, in the data of every future web visit.
Otherwise, you’d have to put in your password over and over again for every page you visited.
Typically, this is done using some sort of Set-Cookie command that is issued by the server during the HTTPS part of the login process:
The Set-Cookie HTTP header above is only visible inside your browser, after the HTTPS layer has been stripped off, because the cookie-setting command itself is shielded by HTTPS, which sounds good.
But this is cold comfort.
Every future request from your browser, whether sent in plain HTTP or in shrouded HTTPS form, includes that exact cookie data in its outgoing request headers, so that the server can tell that it is you coming back to the site:
In other words, HTTPS protection on the login page is little use if the site then reverts to HTTP and leaves your account’s authentication token open to hijack thereafter.
The service providers who went in for this half-baked protection knew this perfectly well, but did it anyway.
As you can imagine, the fact that some of the biggest and most influential online services behaved this way wasn’t much of an incentive to smaller businesses to move to HTTPS.
In two words, given that avoiding HTTPS could save companies time and money, “Why bother?”
All that changed in 2010, when a Firefox plugin called Firesheep was released by a security researcher and activist called Eric Butler.
Because sniffing out and abusing web authentication tokens required technical skills that many users didn’t have, Butler created a point-and-click tool that you could load into Firefox (on your own computer – this wasn’t a tool you could embed in a website to attack someone else remotely) in order to learn about the importance of HTTPS.
Firesheep would not only alert you to the presence of other people’s cookies in the ether, but also grab them for you if so requested, and offer to set up “cloned” sessions automatically in those users’ names.
(Doing this without their permission was illegal in almost every country, of course, but by “attacking” your own social media accounts, you would quickly appreciate the clear and present danger posed by half-baked HTTPS implementations.)
In Butler’s own words at the time:
When logging into a website you usually start by submitting your username and password. The server then checks to see if an account matching this information exists and if so, replies back to you with a “cookie” which is used by your browser for all subsequent requests.
It’s extremely common [in 2010] for websites to protect your password by encrypting the initial login, but surprisingly uncommon for websites to encrypt everything else. This leaves the cookie (and the user) vulnerable. HTTP session hijacking (sometimes called “sidejacking”) is when an attacker gets a hold of a user’s cookie, allowing them to do anything the user can do on a particular website. […]
This is a widely known problem that has been talked about to death, yet very popular websites continue to fail at protecting their users. The only effective fix for this problem is full end-to-end encryption, known on the web as HTTPS or SSL. Facebook is constantly rolling out new “privacy” features in an endless attempt to quell the screams of unhappy users, but what’s the point when someone can just take over an account entirely? Twitter forced all third party developers to use OAuth then immediately released (and promoted) a new version of their insecure website. When it comes to user privacy, SSL is the elephant in the room.
Firesheep certainly put the cat amongst the pigeons, or perhaps the fox among the sheep.
Whether Firesheep was the major catalyst for the widespread adoption of HTTPS that followed is hard to tell, but the outcome was positive nevertheless.
Many major web properties, notably Facebook and Twitter, whom Butler called out by name in the remarks we quoted above, started offering full-time HTTPS by default within about 18 months of Firesheep making headlines.
Smaller businesses, charities and hobby sites didn’t follow suit immediately, however, for a variety of reasons we discussed in Part 1, notably the fact that HTTPS connections require each server to produce a digital certificate to vouch for itself every time anyone connects.
Creating a new certificate for your own website takes seconds and costs nothing, but before any browser will accept it, the certificate needs to be sent off to a so-called CA, or certificate authority, which is expected to carry out at least a basic check to ensure that you really do control the website you’re claiming to operate, and then to add a digital signature of their own to your certificate.
Without this “certificate to vouch for your certificate”, all modern browsers will visibly and offputtingly baulk at accepting it, given that anyone can put any name they like into a new certificate:
Until about 2014, this certificate certification process could be both expensive and troublesome, especially for hobbyists and small businesses.
Some CAs charged hundreds of dollars a year for every certificate they signed, and many businesses had more than one domain name in use.
Just as troublesome were the often bureaucratic procedures that made it hard to automate certificate renewals, leading to embarrassing outages for customers and visitors if certificates weren’t renewed in time. (Browsers flag up expired certificates with dramatic warnings similar to the one above.)
But a spirited effort by cybersecurity activists, backed by a practical and effective non-profit project called Let’s Encrypt, led to a set of human-friendly tools and protocols that made it easy and free to activate HTTPS on almost any website.
Let’s Encrypt has now been joined by numerous similar CA services that are not only free but also easy to use via open-source tools that can automatically renew your certificates for as long as you have control over the websites concerned.
Many if not most web hosting services now use these tools to issue and maintain HTTPS certificates for you as part of your service, so you don’t have to.
These days, few websites use HTTP at all, except perhaps to redirect old links that still start with http:// to their https:// equivalents.
If you’re still offering content for download via HTTP, don’t!
The availability of free, signed certificates that all browsers will accept by default, and the ease with which you can renew these certificates automatically, means that there’s no reason not to offer HTTPS connections to your customers and visitors.
Search engines will mark down your site significantly if you don’t support HTTPS, to encourage you to embrace it in order to make unexpected surveillance and tampering less likely.
If your browser offers an “HTTPS everywhere” option that blocks HTTP-only sites altogether, consider enabling it if it isn’t on by default.
This stops you wandering onto unencrypted sites by mistake, because it tells your browser not merely to prefer https:// in the address bar, but to switch to it anyway, even if you click on an old link that still goes to an http:// URL.
Don’t forget that HTTPS secures the connection, not the site itself.
Because almost everyone is using HTTPS these days, and almost every hosted website runs over HTTPS by default, that means almost all cybercriminals are using HTTPS, too.
The TLS encryption tunnel doesn’t vouch for the quality of the site you’re connecting to, because certificate authorities are there to sign certificates, not to review the material hosted by the site itself.
HTTPS is a technology that deliberately makes it hard for snoops and stalkers to spy on anything and everything you do in your browser, and makes it equally hard for them to manipulate the content you see after it has left the server you’re visiting.
But HTTPS isn’t an automated tool for judging the quality and truthfulness of the site’s content – that remains a task that requires an expert human touch… tools alone don’t cut it!
Why not ask how SolCyber can help you do cybersecurity in the most human-friendly way? Don’t get stuck behind an ever-expanding convoy of security tools that leave you at the whim of policies and procedures that are dictated by the tools, even though they don’t suit your IT team, your colleagues, or your customers!
Paul Ducklin is a respected expert with more than 30 years of experience as a programmer, reverser, researcher and educator in the cybersecurity industry. Duck, as he is known, is also a globally respected writer, presenter and podcaster with an unmatched knack for explaining even the most complex technical issues in plain English. Read, learn, enjoy!
Featured image of padlocked door by Jornada Produtora via Unsplash.
By subscribing you agree to our Privacy Policy and provide consent to receive updates from our company.
