Everything that Google can show us is roughly just 4% of the internet, the remaining 96% lies in depths that the majority of us have never been to.

These are rough and widely cited numbers that are generally considered to be approximate, but the true depths of the internet is unknown till this date.

In this five-part series I decided to take a deep dive and explore the internet beyond the surface. Along the way I will debunk myths, dissect the layers, and explore the depths and the dangers lurking behind them.

Before we go beyond the internet it’s necessary to understand the foundations it stands upon, so in this part I will walk you through an overview of the fundamentals before we get to the depths.

The Internet

To begin this journey we need to first understand what the internet is and what constitutes it.

At its core the internet is just a network of computers, where a network is generally a group of computers that share resources and data. But this would be a vague definition without defining what type of network it is. To start with, there are various types of networks the common ones being Personal Area Network (eg: Bluetooth, hotspot), Local Area Network (eg: home WiFi, office WiFi), Metropolitan Area Network (eg: city-wide WiFi, cable TV) and Wide Area Network (eg: the internet). The major difference among these network types is the area they cover, from very short range connections in PAN to connections spanning wide across countries in WAN.

Generally speaking the internet is an interconnected network of WANs, meaning multiple WAN networks spanning across countries are connected to each other. These consist of the multiple ISPs that transport your data on a global scale.

So essentially it’s just a couple of computers connected to each other on a global scale.

Wait, so the internet is just the infrastructure? I thought Google was the internet..

This is a common misconception. The words internet and web are used interchangeably a lot, even though both are connected to one another they are two different things.

The Internet Vs The Web

To understand this and debunk this myth, we need to tweak the definition of the internet a little bit. While it’s generally okay to say that the internet is the infrastructure connecting all of us together, it’s way more than just the connections. The internet also helps us communicate with the help of various protocols built into it — like SMTP for mail, HTTP for web pages, and FTP for file transfer. Each protocol supports one or the other service running on the internet, so the internet is not just the infrastructure — it’s the protocols and the infrastructure combined to make communication possible. One of the core protocols running the internet is IP or the Internet Protocol, which basically lets us identify devices using a 32-bit address represented as four octets in dotted notation (192.168.1.1) called an IP address or specifically IPv4 and communicate with that device. In short IP is the postal service of the internet, delivering letters or in this case packets to their destination. It’s not the only protocol involved in delivering packets but it is the core one.

So, what is the web?

Like I said the internet has many services running on it and one such service is the web or world wide web. Each time you visit Facebook or a similar site it’s just an HTML document sitting on a server.

The web or World Wide Web is just a global collection of these documents or web pages, interconnected through hyperlinks — much like a spider’s web where every thread leads to another. In order to access the web and get to a webpage you need the help of a few protocols and pieces of software. When you access a webpage you do so by sending a request to the IP address of the server holding that webpage for this the HTTP (HyperText Transfer Protocol) or HTTPS (HyperText Transfer Protocol Secure) is used, which deals with the transfer of hypertext documents.

But that’s not enough we need to render the syntax into a human readable format and for this we need a software, and that software is your browser. The browser (eg: Chrome, Firefox, Edge) manages the underlying protocols like HTTP, helping you send and receive HTML documents. But there is still a problem from what we have discussed it’s clear you can only access a webpage if you know the IP address or domain name, which is a human readable name assigned to an IP to make it easier to access, just like you can only send a letter if you know the destination address.

This is where search engines come in. A search engine (eg: Google, Bing, Yandex) utilizes the core concept of HTML which is hyperlinks, to gather HTML documents and make them accessible as an index. For this the search engine uses an automated program called web crawlers or spiders. They crawl the web to collect HTML documents and record their presence in an index which is then accessed by users like you and me.

But how does it do that?

Before I explain it I want you to go to Wikipedia and keep clicking the first link you see — for example you searched for a word, the Wikipedia page for that word showed another hyperlink for another word, and you keep clicking until there are no more hyperlinks. Soon you will realise it’s bedtime and you just got stuck in a rabbit hole.

Alternatively you can visit thewikigame:

Spiders on the other hand are good at navigating these rabbit holes (or World Wide Web) — they get to a webpage, extract all the hyperlinks, and do the same with each of those webpages while recording or indexing every new webpage they find. So when you search something like “TOR” it searches its database for all the webpages it has with the keyword TOR in their content and returns you the results.

This was essentially the web in an oversimplified way. The web or the world wide web is way more complex with way more protocols and services running in the background, but this just laid the foundation for what we are about to dive into.

Now, remember how I said spiders can crawl through all the webpages on the internet? Well there is a problem with this. Let’s say a web page has links to confidential documents meant for authorized users only, and then a spider comes across this and indexes those confidential documents into the public internet — keep in mind that before this they weren’t publicly available they were meant to be confidential.

To avoid this, just like in real life, the internet has some regulations. Protocols are one example of these rules, another being robots.txt, a text file that dictates what a crawler can access and what it cannot. Well just like a real life maverick, bots or crawlers can decide to ignore it, in which case they cross the legal boundaries of the internet becoming malicious bots rather than ordinary web crawlers.

This is also where ironically the curious or malicious minds can find an advantage — robots.txt is publicly accessible, meaning anyone can pull it up and get a basic idea of what a website is trying to hide or what it has accidentally exposed, like the confidential files we just talked about.

Try it yourself — go to your favourite site and check its robots.txt, though keep in mind some sites don’t have one:

This is why defence in depth matters. There needs to be multiple layers of defence to stop a crawler or an adversary who ignored or misused the robots.txt.

A few other ways to protect content from crawlers apart from robots.txt is putting it behind a login page. Crawlers cannot perform much beyond scraping hyperlinks and content — they cannot go beyond a login page. This means anything behind a login page is likely not indexed, though in some instances it can be partially indexed, like a public Facebook account.

If you go to your Facebook settings and navigate to the Privacy tab, there is a section called “How People Find And Contact You.” Within this section there is an option to allow or block search engines from indexing your profile disabling it makes your profile unindexed while leaving it enabled leaves it open to web crawlers.

Another way to prevent crawlers and malicious bots is using CAPTCHAs. Even though crawlers are smart, they aren’t human enough to solve a CAPTCHA — though in today’s world crawlers are far more complex, and in a way mutated, with LLMs and AI scraping webpages at scale to train their models.

This just proves one point even though crawlers can scrape the internet, due to the restrictions discussed above they cannot index everything. Remember anything indexed is publicly available, which means there are webpages that have never existed in the public internet.

Wait, public internet? I thought everything on the internet was public?

No, not everything is public. There are parts of the internet a crawler cannot index because it never reaches them, for multiple reasons. A few as discussed above are login pages, CAPTCHAs, firewalls, or simply because no other webpage has ever linked to them.

Wait, how is that possible?

Well if you create a webpage on your local machine and host it with nginx, unless some webpage has a backlink pointing to your IP address and port, crawlers don’t have access to it.

Oversimplified, but you get the idea there are webpages out there that Google cannot access simply because it never finds them.

An important thing to note here is that while Google may not be able to find them due to how its crawlers work, there are search engines specifically built for this purpose. These scan public networks and ISPs to find open ports and services exposed on the internet .

Try it yourself — find your public IP address, not your local or private one (eg: 192.168.X.X, 172.16.X.X, or 10.X.X.X) and search for it. See what comes up. Again if your ISP uses firewalls to block such connections it simply means your IP is not publicly exposed.

This is where I would like to introduce the layers of the web, where each layer is defined by how you access the services over internet.

Layers of the Web

This iceberg is essentially the web. The tip is the surface web, the one we all use and have access to, what I referred to as the public internet. Everything Google and other similar search engines can access is the surface web, accounting for roughly close to 4% of the whole internet. Then we get deeper into the deep web and the dark web an area filled with misconceptions and myths, so hang tight as we explore the deeper layers.

To access the surface layer all you need is a surface web search engine like Google. But as we discussed it cannot access content locked behind login screens or content restricted to crawlers — this includes your private social media accounts, admin dashboards, your bank account, private emails, and so on. All of this makes up the deep web, taking up roughly 96% of the internet. In simpler words, 96% of the internet is restricted to certain authorized users only for example you are the only one who can access your private Instagram account or your Gmail. These can still be accessed from browsers like Chrome but require the right credentials.

I am sorry to say — if you were hoping for some dark stuff when I said deep web, you were wrong. It’s a popular myth that the deep web requires something like TOR to access, because the deep web is just the unindexed part of the World Wide Web. The only special thing about it is that it has restrictions and is not publicly available.

Now finally to the part we have been waiting for the dark web. It sounds ominous and scary but in reality it’s quite different. While the dark web is widely used by criminals it is not an illegal association like the movies make it out to be. The dark web is essentially a subset of the deep web out of that 96%, a portion is occupied by the dark web, consisting of content that is not merely private like the deep web but practically hidden it cannot be accessed directly from the public internet. To understand this I have to introduce a technology that pioneered privacy and anonymization on the internet, the very technology that essentially gave birth to the dark web so hang tight for the next part where we understand TOR and how it gave birth to the dark web.

Until next time, stay safe…