urlscan.io Blog


Product Updates for 2021

This post will be a recap of new features we launched in 2021, covering our community platform and our commercial products. There will be a separate post with our 2022 product roadmap later.

Scanning Engine v2

As one of our biggest projects in 2021, we sat down and rewrote our scanning engine from scratch. The result was the Scanning Engine v2 which can be used for a multitude of purposes. The most important use-case is still the regular scanning of URLs submitted through urlscan.io without any visible changes to the user. Other use-cases include the Live Scanning feature and different internal scanning tasks that can now all be covered by the same codebase. These changes have helped us deploy our engine much more quickly with just a few lines of infrastructure definition. The new engine is also much more modular, allowing users to define whether to store data to backend storages for example. The scanner has a vast array of options now that can be changed at scan time and a modular architecture that allows us to run it with different backend modules and different connectivity options.

(Read the rest of this post...)


Announcing the urlscan.io Blog

Today we’re announcing the urlscan.io Blog. This blog covers announcements, product news, tutorials, and service incidents.

  • Announcements: Big upcoming changes that might have an impact on how you use the platform.
  • Product News: Recent feature additions and how to use them.
  • Tutorials: In-Depth guides on how to make the most use of our platform.
  • Incidents: Documentation incidents and outages in our service.

There are two main categories of posts: Updates and Incident Reports. Updates will appear on the frontpage, while Incident Reports can be viewed on a separate page. The RSS feed and email newsletter will always contain both categories of news items.

You can subscribe to updates via RSS or via our email newsletter.

We will continue to Tweet news via our official Twitter account as well.


Search API Limitations for Free Users

We recently had to make some changes to how our Search API behaves for free and anonymous users. Anonymous users are users without a user-account, and free users are users which have a user-account, but which are not on any of our paid tiers.

We had to make these changes to ensure the continued stability of our Search API. Requiring user-signup allows us to better spot malicious behavior or talk to legitimate users about “bad” queries which they are not aware of. Thank you for your understanding.

Changes to our Search API

  1. Regex search is not available anymore to users without a user-account.
  2. Leading Wildcard search is only available to users on the urlscan Professional and urlscan Enterprise plans.
  3. The maximum number of search results returned by the Search API is limited:
    • Anon Users: 100 results max
    • Free Users: 1000 results max
    • Paid Users: 10000 results max

Changes to the "Structurally Similar Pages" Feature

The Structurally Similar Pages feature on the result page is not visible to users without a user-account. This page was heavily scraped, so we had to make this change. If you are interested in getting API-access to the Structurally Similar Pages feature, check out our paid tiers!



API Quotas & Rate-Limits are active

The daily quotas and per-minute/per-hour rate-limits for our API plans are active as of today. To see how much quota you’re using, check out your user dashboard. If you are submitting scans as private because your tool or integration doesn’t support choosing the scan visibility, you can override the behaviour in the user settings. Go to User → Settings & API → Change Settings and then change the default visibility and select “Enforce visibility.”


Summer 2020 - Lockdown Edition

This is the first release in a long time and contains many major feature improvements and additions.
We're happy to welcome Joe Security as a new sponsor for the urlscan.io community service!

Commercial Products

We have finalised our commercial Product lineup. If your organisation needs to perform a large number of API requests, such as API searches or scan submissions, you can subscribe to one of our API plans. There is a generous Free tier available. Check out our Products page and get in touch with our Sales team today!

API Documentation

We have overhauled our API documentation and made it more explicit in many areas. There is a comprehensive best-practices guide for working with our API which is especially important if you working on an automated integration with urlscan.io. The current APIv1 won't receive any major or breaking changes from now on, it will eventually be superseded by a more powerful APIv2.

API Quotas

API rate limits and quotas are accounted for for all API actions. The API documentation contains information how to work with the quotas. Users should use their API keys for all API actions so these can be accounted for properly. You can view your current quotas and how much you used on your user dashboard and on the Quotas page.

Unlisted Scans

This release also introduces the new Unlisted scan type. Make sure you understand the difference between Public, Unlisted and Private scans and when to use which one as outlined on our API documentation page.

Teams

Teams allow you to collaborate and share private and unlisted scans with team-members. Teams have their own quotas. For commercial users, you can set up a Team Subscription so all of your users benefit from the additional features of that subscription.

Small Features & Improvements

Logged-in users get access to a couple of new features:

  • Settings: Default scan visibility and visibility enforcement.
  • Quotas: Users can view their personal quotas from their dashboards.
  • Teams: Users can create teams and invite other urlscan.io users.
  • Disable API keys: You can disable API keys, e.g. if they were compromised.

These small improvements were made:

  • We try to detect scans containing PII and will automatically restrict visibility if we detect it.
  • The Live scans page has been improved to show more relevant scans.
  • You can search your own scans in the regular search UI / API via
    user:me
    .
  • The submit dialog remembers your last visibility.
  • The search UI remembers your last detail settings.

Breaking Changes

  • Scan API Public URL submissions which we believe contain PII (such as email addresses) will automatically be reclassified as Unlisted.
  • Scan API The
    public
    option was deprecated in favor of the explicity
    visibility
    option.
  • Search API The
    offset
    option was deprecated. If you need to iterate over older scans please use the
    date
    field.


Subscribe via RSS