urlscan.io Search API Reference v1
Last updated: 2022-04-20

Our Search API & UI allows you to find archived scans of URLs on urlscan.io. This page is a reference for the available fields that can be used to query the API. Please see explanations about the field types and visibility below!

Query String Syntax and General Instructions

• Search requests (through the UI or API) are subject to your individual Search API Quotas. Make sure to use your API key.
• The query field uses the ElasticSearch Query String to search for results.
• All queries are run in filter mode, sorted by date with the more recent scans first. There is no scoring of search results.
• You can group and concatenate search-terms with brackets
  ( )
  ,
  AND
  ,
  OR
  , and
  NOT
  . The default operator is
  AND
  .
• You can concatenate terms within a group, e.g.
  page.domain:(foo.com OR bar.com)
  .
• Always use the field names of the fields you want to search. Wildcards for the field-name are not supported! Field names are case sensitive!
• Always escape reserved characters with backslash:
  + - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \ /
• Limit the time-range if possible using
  date
  , e.g.
  date:>now-7d
  or
  date:>now-1y
  .
• The
  date
  allows relative queries like
  date:>now-7d
  or range-queries like
  date:[2020-01-01 TO 2020-02-01]
  or both combined.
• You can only use leading wildcard searches and regular expression searches on supported fields, and only as a signed-in user.
• Everything is indexed as lowercase, even if the Search API returns values in a case-preserving manner.
• Regular expressions are always anchored to beginning/end of the tokens (implicit ^ and $). Make sure to prefix/suffix with
  .*
  to match infix strings.
• Domain fields contain the whole domain and each smaller domain component, i.e.
  domain
  can be searched by google.com which will find hits for www.google.com

Recent Changes

April 20, 2022 - Search Index Version v1.2

• Changed Behaviour:
  • date: Contains the timestamp for when the scan finished rather than when the scan was submitted.
  • The
    indexed_at
    field was removed. Please rely on the
    task.time
    field.
  • The
    submitter.country
    field is now lowercase and you have to search it via lowercase queries!
  • Searching fields that your user does not have access to will result in a HTTP/400 error from the search API. Previously these searches simply didn't return any results.
• keyword RE: Fields marked with this symbol allow fast and efficient search using infix queries and regular expressions. If you try to perform an infix or regex query with a leading wildcard on a field which does not support it, we will try to rewrite the query to use the equivalent keyword RE field if it exists.
• Additional Fields: We have added additional fields for searching. Some are new top-level fields and others are available under existing parent fields. New top-level fields include content dom frames links text verdicts
• Implementation guidance: Make sure your code can handle missing fields in the search results. While fields are strongly typed, they can absent in certain records.
• Deprecated behavior: The screenshot and result fields in the result entry for each scan are deprecated and should not be used any more. The corresponding URLs can be constructed using the _id. These fields will be removed in future versions of this API!
• Improved Documentation: New fields are highlighted in the list below.

Field Type Legend

The available search fields can have different types, which dictate how they can be searched. It is important to understand the limits of each type to fully utilise our Search API.

• — Field can be searched and is returned in the search results.
• — Field can be searched, but the original value is not returned in the search results.
• keyword — Field is analysed as one keyword, use an exact value or a trailing wildcard search to search. Indexed as lowercase.
• keyword RE — Field can be searched by regular expression and leading wildcard. Indexed as lowercase.
• text — Field is analysed as text, broken into multiple tokens (e.g. split on slash in the URL). Phrase search with quotes possible. Indexed as lowercase.
• date — Field is analysed as date, allowing range-queries and date math, e.g.
  date:>now-24h
  .
• ip — Search by an IPv4 or IPv6, either using an exact IP or a subnet definition like
  ip:8.8.8.8\/24
  .
• domain — Search by a domain or parent domain. You can search for
  www.foobar.com
  or just
  foobar.com
  and it will both find scans for www.foobar.com.
• integer — Allows searching by exact value, range, or threshold, e.g.
  stats.uniqIPs:>5
  .

Searchable Fields