urlscan.io Blog

Internet-Wide IPv4 Scan Data

We are now offering raw download access to the following datasets to interested customers:

  • Weekly Internet-wide scans of the whole IPv4 space on ports tcp/80 and tcp/443.
  • JSON output containing TLS certificates and HTTP responses.
  • More than 200GB of compressed raw data available per week.
  • More than 40 million HTTP responses on tcp/80 and more than 35 million on tcp/443.

The HTTP scan results contain the following pieces of data and metadata:

  • Observation timestamp
  • GeoIP and ASN network annotation for the IP which was scanned
  • HTTP Status code & string
  • HTTP Response Headers & content
  • Response Body Size & SHA256
  • Response Body (up to 64kB)
  • TLS Certificate Subject, Issuer, Serial Number and MD5/SHA1/SHA256 fingerprints

The certificate scan results contain certificates that responded to HTTPS requests:

  • Raw unparsed TLS certificate data
  • Parsed certificate data

Sample Data

These samples are 10,000 lines from the respective datasets and should give you an idea what’s contained within the data.

HTTP Scans on tcp/80 - HTTP Responses

HTTP Scans on tcp/443 - HTTP Responses

HTTP Scans on tcp/443 - TLS Certificates

Access to Data

If you are interested in access to these data-sets then please reach out to us at support@urlscan.io. We offer free 30-day trial access.