urlscan.io Search API Reference v1
Last updated: 2021-01-26
Our Search API & UI allows you to find archived scans of URLs on urlscan.io. This page is a reference for the available fields that can be used to query the API. Please see explanations about the field types and visibility below!
Query String Help
- Search requests (either through the UI or API) are subject to your individual API Quotas.
- The query field uses the ElasticSearch Query String to search for results.
- All queries are run in filter mode, sorted by date descending. There is no scoring of search results.
- Refer to the documentation for advanced queries such as wildcard, regex, boolean operators, fuzzy searches, etc.
- You can group and concatenate search-terms with brackets ( ),AND,OR, andNOT. The default operator isAND.
- You can concatenate terms within a group, e.g. page.domain:(foo.com OR bar.com).
- Always use the field names of the fields you want to search. Wildcards for the field-name are not supported!
- Always escape reserved characters with backslash: + - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \ /
- Always limit the time-range if possible using date, e.g.date:>now-7dordate:>now-1y.
- You can use wildcard (leading wildcard only on urlscan Pro) and regex search on almost any field.
- Regexes are always anchored to beginning/end of the tokens.
- The dateallows relative queries likedate:>now-7dor range-queries likedate:[2020-01-01 TO 2020-02-01]or both combined.
- Domain fields contain the whole domain and each smaller domain component, i.e. domaincan be searched by google.com which will include www.google.com
Supported Fields
The following fields are available in the Search API. Fields with a icon can be searched and are shown in the result. Fields with a can be searched as well, but their original value is not returned in the Search API results.
| Field Name (+ Aliases) | Type | Plan | Notes / Possible Values | |
|---|---|---|---|---|
| task.time (date) | date | When the URL was tasked | ||
| indexedAt | date | Time the scan was last indexed in the search index | ||
| asn | keyword | Any of the AS that were contacted (e.g. AS123) | ||
| asnname | text | Any of the AS Names that were contacted | ||
| asnname.keyword | keyword | Any of the AS Names that were contacted (analysed as keyword) | ||
| country | keyword | ISO 3166-1 2-letter country code of any country that was contacted | ||
| domain | domain | Any domain and subdomain that was contacted | ||
| filename (url) | text | Any URL that was requested | ||
| filename.keyword | keyword | Any URL that was requested | ||
| hash (sha256) | keyword | Any SHA256 hash of any HTTP response | ||
| ip | ip | Any IP that was contacted | ||
| server | keyword | Any HTTP "Server" header of subrequests | ||
| page.asn | keyword | AS Number of the website | ||
| page.asnname | text | Name of the main AS of the website | ||
| page.city | keyword | Primary IP GeoIP City information | ||
| page.country | keyword | Primary IP GeoIP Country (ISO 3166-1 2-letter country code) | ||
| page.domain | domain | Primary Domain (Analysed as all levels of parent domains) | ||
| page.domain.keyword | keyword | Primary Domain (Analysed as keyword) | ||
| page.ip | ip | Primary IP | ||
| page.ptr | domain | DNS PTR record of primary IP | ||
| page.server | text | HTTP "Server" header of primary request | ||
| page.status | keyword | HTTP status code of primary request response | ||
| page.url | text | URL of the primary page (after redirection) | ||
| page.url.keyword | keyword | URL of the primary page (after redirection, analysed as keyword) | ||
| stats.dataLength | integer | Data size of all subresources | ||
| stats.encodedDataLength | integer | Transfer size of all subresources | ||
| stats.requests | short | Number of subrequests | ||
| stats.uniqCountries | short | Number of unique countries contacted | ||
| stats.uniqIPs | short | Number of unique IPs contacted | ||
| task.domain | domain | Domain of the tasked URL | ||
| task.domain.keyword | keyword | Domain of the tasked URL (analysed as keyword) | ||
| task.method | keyword | Can be manual, api or automatic | ||
| task.source | keyword | Examples: phishtank or certstream-suspicious | ||
| task.tags | keyword | Custom tags supplied during scan submission | ||
| task.url | text | The original URL that was tasked | ||
| task.url.keyword | keyword | The original URL that was tasked (analysed as keyword) | ||
| task.uuid | keyword | The unique UUID of the scan | ||
| task.visibility | keyword | Can be one of public, unlisted or private | ||
| user | virtual | Scans submitted by yourself (Can only be me) | ||
| team | virtual | Scans submitted by any of your teams (Can only be me) | ||
| apikey | virtual | Scans submitted using one of your API keys (Can only be me) | ||
| submitter.country | keyword | urlscan Pro | GeoIP country of the submission IP (ISO 3166-1 2-letter country code) | |
| brand.country | keyword | urlscan Pro | ISO 3166-1 2-letter country code of the brand | |
| brand.key | keyword | urlscan Pro | Unique key of the brand | |
| brand.name | text | urlscan Pro | Name of the brand | |
| brand.vertical | text | urlscan Pro | Industry vertical of the brand, e.g. "banking" |
Field Type Help
The available search fields can have different types, which dictate how they can be searched. It is important to understand the limits of each type to fully utilise our Search API.
- keyword — Field is analysed as one keyword, use the exact same value or a prefix search to search.
- text — Field is analysed as text, broken into multiple tokens (e.g. split on slash in the URL).
- date — Field is analysed as date, allowing range-queries and date math, e.g. date:>now-24h.
- ip — Search by an IPv4 or IPv6, either using an exact IP or a subnet definition like ip:8.8.8.8\/24.
- domain — Search by a domain or parent domain. You can
search for www.foobar.comor justfoobar.comand it will both find scans for www.foobar.com.
- integer / short — Allows searchring by range, e.g. stats.uniqIPs:>5.