This post will be a recap of new features we launched in 2021, covering our
community platform and our commercial products. There will be a separate post
with our 2022 product roadmap later.
Scanning Engine v2
As one of our biggest projects in 2021, we sat down and rewrote our scanning
engine from scratch. The result was the Scanning Engine v2 which can be used
for a multitude of purposes. The most important use-case is still the regular
scanning of URLs submitted through urlscan.io without any visible changes to
the user. Other use-cases include the Live Scanning feature and different
internal scanning tasks that can now all be covered by the same codebase. These
changes have helped us deploy our engine much more quickly with just a few
lines of infrastructure definition. The new engine is also much more modular,
allowing users to define whether to store data to backend storages for example.
The scanner has a vast array of options now that can be changed at scan time
and a modular architecture that allows us to run it with different backend
modules and different connectivity options.
→ Read the rest of this post...
Today we’re announcing the urlscan.io Blog. This blog covers announcements, product news, tutorials, and service incidents.
-
Announcements: Big upcoming changes that might have an impact on how you use the platform.
-
Product News: Recent feature additions and how to use them.
-
Tutorials: In-Depth guides on how to make the most use of our platform.
-
Incidents: Documentation incidents and outages in our service.
There are two main categories of posts: Updates and Incident
Reports. Updates will appear on the frontpage, while Incident Reports can
be viewed on a separate
page. The RSS feed and email newsletter will always contain both
categories of news items.
You can subscribe to updates via RSS or via our email
newsletter.
We will continue to Tweet news via our official Twitter account as well.
We recently had to make some changes to how our Search API behaves for free and
anonymous users. Anonymous users are users without a user-account, and free
users are users which have a user-account, but which are not on any of our paid
tiers.
We had to make these changes to ensure the continued stability of our Search
API. Requiring user-signup allows us to better spot malicious behavior or talk
to legitimate users about “bad” queries which they are not aware of. Thank you
for your understanding.
Changes to our Search API
- Regex search is not available anymore to users without a user-account.
- Leading Wildcard search is only available to users on the urlscan Professional and urlscan Enterprise plans.
- The maximum number of search results returned by the Search API is limited:
- Anon Users: 100 results max
- Free Users: 1000 results max
- Paid Users: 10000 results max
Changes to the "Structurally Similar Pages" Feature
The Structurally Similar Pages feature on the result page is not visible
to users without a user-account. This page was heavily scraped, so we had to
make this change. If you are interested in getting API-access to the
Structurally Similar Pages feature, check out our paid
tiers!
We have restructured and improved our documentation, you can see all
the topics under the new Documentation page. We
have also created an Search API reference for working with our Search API
since this was a topic we frequently received questions about.
The daily quotas and per-minute/per-hour rate-limits for our API plans are
active as of today. To see how much quota you’re using, check out your user
dashboard. If you are submitting scans as private because your tool or
integration doesn’t support choosing the scan visibility, you can override the
behaviour in the user settings. Go to User → Settings & API → Change Settings
and then change the default visibility and select “Enforce visibility.”
This is the first release in a long time and contains many major
feature improvements and additions.
We're happy to welcome Joe Security as a new
sponsor for the urlscan.io community service!
Commercial Products
We have finalised our commercial Product lineup. If your
organisation needs to perform a large number of API requests, such as
API searches or scan submissions, you can subscribe to one of our API
plans. There is a generous Free tier available. Check out our
Products page and get in touch with our
Sales team today!
API Documentation
We have overhauled our API documentation and made it more explicit
in many areas. There is a comprehensive best-practices guide for
working with our API which is especially important if you working on an
automated integration with urlscan.io. The current APIv1 won't receive
any major or breaking changes from now on, it will eventually be
superseded by a more powerful APIv2.
API Quotas
API rate limits and quotas are accounted for for all API actions.
The API documentation contains
information how to work with the quotas. Users should use their API
keys for all API actions so these can be accounted for properly.
You can view your current quotas and how much you used on your user
dashboard and on the Quotas page.
Unlisted Scans
This release also introduces the new Unlisted scan type. Make sure you understand the difference
between Public, Unlisted and Private scans and when to use which one as outlined on
our API documentation page.
Teams
Teams allow you to collaborate and share private and unlisted scans
with team-members. Teams have their own quotas. For commercial users,
you can set up a Team Subscription so all of your users benefit
from the additional features of that subscription.
Small Features & Improvements
Logged-in users get access to a couple of new features:
- Settings: Default scan visibility and visibility enforcement.
- Quotas: Users can view their personal quotas from their dashboards.
- Teams: Users can create teams and invite other urlscan.io users.
- Disable API keys: You can disable API keys, e.g. if they were compromised.
These small improvements were made:
- We try to detect scans containing PII and will automatically restrict visibility if we detect it.
- The Live scans page has been improved to show more relevant scans.
- You can search your own scans in the regular search UI / API via
user:me
.
- The submit dialog remembers your last visibility.
- The search UI remembers your last detail settings.
Breaking Changes
