If you’re not sick of hearing it yet: Here’s to a happy new year from all of us at urlscan.io!
We wanted to take the opportunity to revisit major changes that launched in 2022 and to give you a glimpse of our 2023 roadmap at the same time. Some of the things we have worked on in 2022 represent the foundation for new products due to launch over the next quarters.
Today we are officially launching our real-time feed and search index of newly observed hostnames and domains on urlscan Pro. This is a huge step forward since it will allow customers to proactively look for new domains and hostnames that might be of interest to them, even if these hostnames were not previously scanned as a full-blown website through urlscan.io.
We are now offering raw download access to the following datasets to interested customers:
This post talks about the different scan visibilities available on urlscan.io, which visibility you should use for different purposes and how to review your submission results on urlscan.io to detect and prevent inadvertent information leaks.
tl;dr: Understand the different scan visibilities, review your own scans for non-public information, review your automated submission workflows, enforce a maximum scan visibility for your account and work with us to clean non-public data from urlscan.io!
Today marks the last day of major features releases we had planned for Q2. This post will cover the highlights of new functionality in our urlscan Pro platform.
You can now save a search in urlscan Pro to be able to run it again later. On top of that, you can also receive an email alert whenever there are new hits for your saved searches. This allows you to create a number of hunt queries which might only trigger occasionally and automatically receive notifications when there are new hits.
The Search page in urlscan Pro was significantly improved:
In the process of scanning websites, urlscan.io will sometimes encounter file downloads triggered by the website. If we are able to successfully download the file, we will store it, hash it and make it available for downloading by our customers.
To highlight this stream of data, we have created a separate Downloads section which contains the most recent file downloads and highlights the information we store for each downloaded file. There is a dedicated Help Section on Downloads which talks about API use and known limitations of this feature.
The following features were added to the the Live Scanning UI:
As of today, our Live Scanning and Visual Search features are no longer considered Beta. The APIs for these features are now stable and can be used in production use-cases. Customers on our Professional and Enterprise subscription tiers will find API documentation for these features in the urlscan Pro platform.
Visual Search allows users to find historical scans with visually similar screenshots to a scan of interest. This type of feature is also called Content-Based Image Retrieval. Check out the accompanying blog-post to learn more.
Live Scanning allows you to scan websites quickly, from different locations, and with different browser options. Scan results are not automatically saved to urlscan.io, but you can use Store Scan if you want to archive a particular scan result.
Live Scanning is a very versatile capability that can be used for a number of common scenarios, including Reconnaissance, Change Monitoring and Remote File Retrieval.
Today we are launching Visual Search which is a powerful new search feature available through our urlscan Pro - Threat Hunting platform.
Visual Search allows users to find historical scans with visually similar screenshots to a scan of interest. This type of feature is also called Content-Based Image Retrieval. Instead of querying for historical scans using a structured textual query (such as search for a hostname or an IP address), Visual Search uses an existing screenshot image as the query. Visual Search works similar to popular Reverse Image-Search engines like Google’s Search by Image and the TinEye Reverse Image Search. Customers will be able to leverage Visual Search feature to discover previously undetected cases of brand impersonation or similar phishing pages based on the visual appearance of those sites.
Visual Search is available today through the urlscan Pro portal. The feature is currently in Beta until its API is finalized over the next few weeks. Further information about Visual Search is available to customers on the urlscan Pro platform.
Today we are launching a major overhaul to our search index powering our urlscan.io and urlscan Pro platforms. This release will offer new functionality to community and paid users. We have gathered customer feedback and internal use-cases and came up with a list of additional attributes that would be helpful to search on. This post outlines the highlights of new available search attributes. All of the new searchable fields have been integrated in a backward compatible fashion, which means that any search which previously worked on urlscan.io will continue to work.
The full list of searchable fields is available on the Search API Reference page.
We just launched the User Verdicts section on the scan result page which lets users submit comments and verdicts about scans on urlscan.io. These verdicts and comments will be publicly visible to other users and should be used to document malicious, suspicious or legitimate websites. Users can classify scans as legitimate, suspicious or malicious, or they can simply leave a comment. If the scan is targeting or impersonating a specific brand then users can select the brand from a predefined list.
Eventually we will integrate these verdicts into our API and into an overall score for each scan, for now we are simply recording them and making them available through the scan result page.
Currently the User Verdicts are only available to users who signed up to urlscan.io more than a month ago. This is to prevent abuse of the feature.
The existing Report feature is still available and should be used when requesting takedown of a scan, for example when the scan contains private or inappropriate content.
This post will be a recap of new features we launched in 2021, covering our community platform and our commercial products. There will be a separate post with our 2022 product roadmap later.
As one of our biggest projects in 2021, we sat down and rewrote our scanning engine from scratch. The result was the Scanning Engine v2 which can be used for a multitude of purposes. The most important use-case is still the regular scanning of URLs submitted through urlscan.io without any visible changes to the user. Other use-cases include the Live Scanning feature and different internal scanning tasks that can now all be covered by the same codebase. These changes have helped us deploy our engine much more quickly with just a few lines of infrastructure definition. The new engine is also much more modular, allowing users to define whether to store data to backend storages for example. The scanner has a vast array of options now that can be changed at scan time and a modular architecture that allows us to run it with different backend modules and different connectivity options.
Subscribe via RSS
