urlscan.io Blog

This post talks about the different scan visibilities available on urlscan.io, which visibility you should use for different purposes and how to review your submission results on urlscan.io to detect and prevent inadvertent information leaks.

tl;dr: Understand the different scan visibilities, review your own scans for non-public information, review your automated submission workflows, enforce a maximum scan visibility for your account and work with us to clean non-public data from urlscan.io!

Scan Visibilities - Introduction

Every time you submit a URL to urlscan.io you can select the visibility for the scan result. The visibility controls which parties will be able to see the URL you submitted and retrieve the scan results.

• Public means that the scan will be visible on the front page and in the public search results and info pages. It will be visible to any visitor on urlscan.io and search engines as well.
  You should only use Public scans if there are no concerns that the URLs you are submitting contain any personal or proprietary information, either in the URL itself or in the content of the page. This could be because you sourced these URLs from another public data set, or because you discovered these URLs yourself via crawling or keyword monitoring.
• Unlisted means that the scan will not be visible on the public page or search results, but will be visible to customers of the urlscan Pro platform. We only admit customers to urlscan Pro which are either vetted security researchers or reputable corporations.
  You should use Unlisted scans if you think that there might be personal or proprietary information within the websites, but you still want to document the URLs to the audience of urlscan Pro so that they can take action accordingly (automated takedowns, research, improving their products).
• Private means that the scan will only be visible to yourself and not to any of our customers or partners. If you are part of a team account and have the team set as “Active”, then your private scans will also be visible to other team members on that team account.
  You should use Private scans if you don’t want to share the scans you perform with anyone else. The downside is that unique URLs that you might submit will not be seen by anyone else, and potential malicious activity might go unnoticed by the community and downstream security companies on the urlscan Pro platform.

There are different reason for choosing specific visibility levels, and picking the right one very much depends on source of the data you are analysing. Customers might have different streams of URLs they want to analyses with urlscan.io, and each stream might have its unique set of privacy considerations.

We encourage user to use Public or Unlisted scans whenever possible since it helps the whole security community keep track and understand threats rather than siloing that information. But we understand that there are use-cases which don’t allow anything but the Private visibility level.

Reviewing your submission results

Whether you use urlscan.io via the UI or the API you should frequently review your submission results to ensure that you are not submitting inappropriate URLs at any visibility level. To see a list of your own submissions first make sure you are logged in to urlscan.io before executing the following searches:

• Search: All Scans submitted by yourself and your teams
• Search: Public Scans submitted by yourself and your teams

Make sure you understand where submissions are originating from: This might be your employees or your automated tools such as SOAR platforms! You should watch out for submissions for the following types of websites:

• Hosted invoice pages
• DocuSign or other document signing requests
• Google Drive / Dropbox links
• Email unsubscribe links
• Password reset or create links
• Web service, meeting and conference invite links
• URLs including PII (email addresses) or API keys

Setting and enforcing a default visibility

urlscan.io allows you to set a default visibility and even to enforce this as the maximum visibility for all future scans. Both settings can be found in your Settings window on your user dashboard.

Team account owners can change these settings team-wide and have them be applied to every active team member. This is done on the Settings page for the team account.

What we are doing to prevent information leaks

We are aware of the fact that non-public information is being scanned on urlscan.io and are taking a number of steps to mitigate this issue.

• We have domain and URL pattern blocklists in place which prevent scanning of certain websites.
• We have deletion rules in place which delete past and future scans for certain keywords and patterns.
• We have recently made the Scan Visibility setting in our user dashboard more visible and easier to understand.
• We have reached out to customers who we identified as submitting a significant amount of Public scans.
• We allow immediate takedown of single scans via the Report button on each scan page.
• We work with customers and third parties to facilitate bulk-delete via our deletion rules.
• We are reviewing popular third-party integrations such as SOAR tools to ensure they respect the user intent with regards to visibility.

Security Researchers

If you are a security researcher and have discovered a large number of scans with non-public information we would ask that you reach out to security@urlscan.io and work with us to get the offending scans removed and to investigate the source of these scans.