At Bitly, we want to make the internet a safer place. Our Trust & Safety team, responsible for designing, developing, and maintaining the organization’s anti-abuse efforts, strives to ensure that each Bitly link you click is secure, trustworthy, and ready to be shared. The road we take to achieve this is long and complex, involving multiple steps in which you, the customer, play a vital role.
In this article, we’ll provide an overview of our anti-abuse platform. It covers the services, systems, technologies, and actions we employ to advance our preventive measures. Our processes start when a link is shortened on our platform, whether you do that directly, when creating a QR Code, or while building a Bitly Link-in-bio page. From there, we’ll introduce our Threat Detection Service (TDS), followed by the abuse application programming interface (API), and finally, the outcome—or what you’ll see—after clicking a Bitly link.
What happens when you, the customer, create a Bitly link?
The first step to verify a link starts when you give us a URL you wish to shorten, a process we call encoding. The encoding of URLs results in a short URL that looks similar to https://bit.ly/3EcLEX9 (or https://yourbrnd.co/3EcLEX8 if using a custom domain). Once a Bitly link has been created, the destination URL for that short link gets sent to a service we call the Crawler. The Crawler’s purpose is to gather information about the website whose link was shortened, like the title of the web page and the type of content hosted there. This information is then sent to the Threat Detection Service.
Bitly’s Threat Detection Service
Our Threat Detection Service is the Trust & Safety team’s first line of defense. Its purpose is communicating with other services to discover whether a given link threatens our community.
Scanning for threats
The TDS leverages the information received from the Crawler and initiates a scan in order to identify and classify harmful content. Both the Crawler and the TDS scan websites for various types of abuse, such as malicious software and phishing attempts. However, unlike with the Crawler, by the time URLs enter the TDS, they are classified into specific types of abuse, which allows a more targeted and accurate abuse detection scan.
Below you will find two examples of threats the TDS has identified. The first is an example of a website that triggers an automatic download of a file that might be malware.
The second is a phishing website whose goal is to steal a user’s Facebook credentials.
Scanning the website gives us an idea of what content we expect to find there. Is it a safe or unsafe website? If the latter, what makes it dangerous? These are the questions TDS scanners aim to find.
Bitly’s Abuse API
Bitly’s Abuse API is the tool we use to set and read the status of a URL or domain. The Abuse API is considered to be Bitly’s single source of truth regarding a URL’s trustworthiness. It adds a URL to our database if that URL falls into one of our abuse categories. At the moment, three primary sources contribute to setting the status of abusive URLs: the output of the Threat Detection Service, our trusted partners, and certain members of the Bitly organization.
After the Threat Detection Service publishes its results, a message is sent via the Abuse API to store the URL and its abuse-related status. We then narrow down the identified categories to maintain consistency across our services and the quality of our reports.
Besides being used by our own services, our Abuse API is available to some other organizations—our trusted partners. When we receive a submission notifying us of suspected abuse from a trusted partner, we evaluate it to determine whether we should add it to our database of unsafe links.
In addition to our internal detection and our trusted partners, we also have a form that anyone can use to report a link with a harmful URL. We read every one of these submissions, check the reported URL, and add it to our system if we find it to be harmful.
Now we arrive at the last stage of the process, verifying if a URL is safe to open when you click on it. As previously mentioned, the Abuse API’s other use is to read the URL’s trustworthiness status. The whole process ends with our Abuse API deciding whether a URL is safe or not.
What happens when you, the user, click a short link?
Welcome, internet user, to the final step of the lifecycle of a short link—where the magic happens once you click. Clicking these links starts the decoding process, which takes a short link, expands it to its long form, and redirects you to its final destination. While decoding a link, we call the Abuse API to check if it is malicious as described in the previous steps. If it is not identified as harmful our journey is over, and you can enjoy the destination. However, if it is harmful, you will instead be redirected to an informational page explaining that the website isn’t safe and providing instructions on what to do next.
For example, if the website’s abuse type falls into our “warn” category, you’ll see a page warning that our systems have flagged the website. This page, which you can see below, shows the long URL but advises you to not click it. Instead, you should notify the person who sent it to you or contact us using this form if you believe the website has been flagged in error.
Another kind of page will be displayed if the website’s abuse type falls into our “blocked” category. Unlike the previous one, you won’t see the long URL and thus won’t be able to click it, but we’ll still allow you to contact us if you believe it is an error from our side. We redirect to this interstitial webpage when we assess that simply accessing the website could directly harm you, such as a website that will automatically download malware to your computer.
Closing words
The internet is an incredible place to connect, learn, watch, communicate, read (articles like this!), and much more. Unfortunately, a portion of it is misused for nefarious purposes that might negatively affect its users. Bitly wants to help with this problem. Our platform sees millions of links shortened, and way more clicked, every single day. We work hard to ensure that these links and scans are safe, and most importantly, that you can trust that the underlying URL is truthful. Our Trust & Safety team was created to assist in this endeavor. Since its creation, the team has developed multiple systems that keep our platform clean and our users safe.
This article introduced our abuse system and some of the techniques we use to keep our platform free of malicious links. We hope you now have a better understanding of the pipeline a link goes through once a user shortens a link or expands one, as well as how our APIs and services interact with each other. Plus, now you know how you can report links to us to improve our platform. And we thank you for that! After all, it’s up to all of us to make the internet a safer place.