robotstxt.com
Open in
urlscan Pro
2001:4860:4802:36::15
Public Scan
Effective URL: https://robotstxt.com/tester
Submission: On February 14 via api from SG — Scanned from SG
Summary
TLS certificate: Issued by GTS CA 1D4 on February 8th 2024. Valid for: 3 months.
This is the only time robotstxt.com was scanned on urlscan.io!
urlscan.io Verdict: No classification
Domain & IP information
IP Address | AS Autonomous System | ||
---|---|---|---|
1 1 | 35.225.242.216 35.225.242.216 | 396982 (GOOGLE-CL...) (GOOGLE-CLOUD-PLATFORM) | |
1 10 | 2001:4860:480... 2001:4860:4802:36::15 | 15169 (GOOGLE) (GOOGLE) | |
1 4 | 2606:4700::68... 2606:4700::6811:3b8 | 13335 (CLOUDFLAR...) (CLOUDFLARENET) | |
12 | 3 |
ASN396982 (GOOGLE-CLOUD-PLATFORM, US)
PTR: 216.242.225.35.bc.googleusercontent.com
googleextended.com |
Apex Domain Subdomains |
Transfer | |
---|---|---|
10 |
robotstxt.com
1 redirects
robotstxt.com |
24 KB |
4 |
cloudflare.com
1 redirects
challenges.cloudflare.com — Cisco Umbrella Rank: 4410 |
13 KB |
1 |
googleextended.com
1 redirects
googleextended.com |
266 B |
12 | 3 |
Domain | Requested by | |
---|---|---|
10 | robotstxt.com |
1 redirects
robotstxt.com
|
4 | challenges.cloudflare.com |
1 redirects
robotstxt.com
challenges.cloudflare.com |
1 | googleextended.com | 1 redirects |
12 | 3 |
This site contains links to these domains. Also see Links.
Domain |
---|
http.app |
webvitals.dev |
disavowfile.com |
urlparse.com |
seoapi.com |
http.codes |
http.dev |
online.marketing |
seo.services |
fili.com |
Subject Issuer | Validity | Valid | |
---|---|---|---|
robotstxt.com GTS CA 1D4 |
2024-02-08 - 2024-05-08 |
3 months | crt.sh |
challenges.cloudflare.com Cloudflare Inc ECC CA-3 |
2023-08-18 - 2024-08-17 |
a year | crt.sh |
This page contains 3 frames:
Primary Page:
https://robotstxt.com/tester
Frame ID: 4BEDBCF05BBE08C356B631709E0C805F
Requests: 11 HTTP requests in this frame
Frame:
https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/b/turnstile/if/ov2/av0/rcv0/0/xr4je/0x4AAAAAAABGxNgJRPWdLYLw/auto/normal
Frame ID: 5E4867E1B5C8D9A1766B95FE2D388944
Requests: 1 HTTP requests in this frame
Frame:
https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/b/turnstile/if/ov2/av0/rcv0/0/7ux56/0x4AAAAAAABGxNgJRPWdLYLw/auto/normal
Frame ID: 2021ED1D5530B524787F0FA1D52F04D6
Requests: 1 HTTP requests in this frame
Screenshot
Page Title
robots.txt TesterPage URL History Show full URLs
-
http://googleextended.com/
HTTP 301
https://robotstxt.com/?utm_source=googleextended.com HTTP 302
https://robotstxt.com/tester Page URL
Detected technologies
Clipboard.js (Miscellaneous) ExpandDetected patterns
- clipboard(?:-([\d.]+))?(?:\.min)?\.js
Page Statistics
24 Outgoing links
These are links going to different origins than the main page.
Title: HTTP Status Tester new
Search URL Search Domain Scan URL
Title: Cumulative Layout Shift Debugger
Search URL Search Domain Scan URL
Title: Web Vitals Score Card
Search URL Search Domain Scan URL
Title: Disavow File Validator
Search URL Search Domain Scan URL
Title: Disavow File Generator
Search URL Search Domain Scan URL
Title: URL Parse API
Search URL Search Domain Scan URL
Title: Googlebot API
Search URL Search Domain Scan URL
Title: Applebot API
Search URL Search Domain Scan URL
Title: Baiduspider API
Search URL Search Domain Scan URL
Title: Bingbot API
Search URL Search Domain Scan URL
Title: Cốc Cốc Bot API
Search URL Search Domain Scan URL
Title: DuckDuckBot API
Search URL Search Domain Scan URL
Title: Naver Yeti Spider API
Search URL Search Domain Scan URL
Title: PetalBot API
Search URL Search Domain Scan URL
Title: SeznamBot API
Search URL Search Domain Scan URL
Title: Sogou Web Spider API
Search URL Search Domain Scan URL
Title: Yahoo! Slurp API
Search URL Search Domain Scan URL
Title: YandexBot API
Search URL Search Domain Scan URL
Title: HTTP Response API new
Search URL Search Domain Scan URL
Title: HTTP Status Codes
Search URL Search Domain Scan URL
Title: HTTP Headers
Search URL Search Domain Scan URL
Title: Online Marketing Conferences
Search URL Search Domain Scan URL
Title: Improve your SEO!With SEO audits by ex-Google Search engineersseo.services
Search URL Search Domain Scan URL
Title: Fili
Search URL Search Domain Scan URL
Page URL History
This captures the URL locations of the websites, including HTTP redirects and client-side redirects via JavaScript or Meta fields.
-
http://googleextended.com/
HTTP 301
https://robotstxt.com/?utm_source=googleextended.com HTTP 302
https://robotstxt.com/tester Page URL
Redirected requests
There were HTTP redirect chains for the following requests:
Request Chain 3- https://challenges.cloudflare.com/turnstile/v0/api.js?compat=recaptcha HTTP 302
- https://challenges.cloudflare.com/turnstile/v0/b/a990e557/api.js?compat=recaptcha
12 HTTP transactions
Method Protocol |
Resource Path |
Size x-fer |
Type MIME-Type |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
GET H2 |
Primary Request
tester
robotstxt.com/ Redirect Chain
|
55 KB 8 KB |
Document
text/html |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
Redirect headers
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GET H2 |
app.css
robotstxt.com/css/ |
27 KB 6 KB |
Stylesheet
text/css |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GET H2 |
ln.min.js
robotstxt.com/js/ |
1 KB 731 B |
Script
application/javascript |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GET H2 |
a.js
robotstxt.com/ |
1 KB 817 B |
Script
application/javascript |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GET H2 |
api.js
challenges.cloudflare.com/turnstile/v0/b/a990e557/ Redirect Chain
|
38 KB 13 KB |
Script
application/javascript |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
Redirect headers
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GET H2 |
ip.min.js
robotstxt.com/js/ |
3 KB 1 KB |
Script
application/javascript |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GET H2 |
clipboard.min.js
robotstxt.com/js/ |
9 KB 3 KB |
Script
application/javascript |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GET H2 |
app.min.js
robotstxt.com/js/ |
1 KB 622 B |
Script
application/javascript |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GET H2 |
t.js
robotstxt.com/js/ |
14 KB 3 KB |
Script
application/javascript |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GET DATA |
truncated
/ |
185 B 0 |
Image
image/svg+xml |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GET H3 |
normal
challenges.cloudflare.com/cdn-cgi/challenge-platform/h/b/turnstile/if/ov2/av0/rcv0/0/xr4je/0x4AAAAAAABGxNgJRPWdLYLw/auto/ Frame 5E48 |
0 0 |
Document
text/html |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GET H3 |
normal
challenges.cloudflare.com/cdn-cgi/challenge-platform/h/b/turnstile/if/ov2/av0/rcv0/0/7ux56/0x4AAAAAAABGxNgJRPWdLYLw/auto/ Frame 2021 |
0 0 |
Document
text/html |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
POST H2 |
event
robotstxt.com/api/ |
2 B 317 B |
XHR
text/plain |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General
Request headers
Response headers
|
Verdicts & Comments Add Verdict or Comment
32 JavaScript Global Variables
These are the non-standard "global" variables defined on the window object. These can be helpful in identifying possible client-side frameworks and code.
object| 0 object| 1 function| plausible function| linenumbers function| toggleMobileMenu object| grecaptcha object| turnstile function| d function| u function| f function| h function| m function| p function| ClipboardJS object| codeSnippets string| codeSnippetButton object| clipboard function| rotate180 object| alert_title object| alert_message object| download object| ua_input object| title object| copy_directives object| directives_input object| ua_select object| url_input object| button_failed object| button_allowed object| button_blocked object| fetch_form function| freeze0 Cookies
Cookies are little pieces of information stored in the browser of a user. Whenever a user visits the site again, he will also send his cookie values, thus allowing the website to re-identify him even if he changed locations. This is how permanent logins work.
Security Headers
This page lists any security headers set by the main page. If you want to understand what these mean and how to use them, head on over to this page
Header | Value |
---|---|
Strict-Transport-Security | max-age=63072000; includeSubDomains; preload |
X-Content-Type-Options | nosniff |
X-Frame-Options | SAMEORIGIN |
X-Xss-Protection | 1; mode=block |
Indicators
This is a term in the security industry to describe indicators such as IPs, Domains, Hashes, etc. This does not imply that any of these indicate malicious activity.
challenges.cloudflare.com
googleextended.com
robotstxt.com
2001:4860:4802:36::15
2606:4700::6811:3b8
35.225.242.216
1f028e9cf42200eec92ddee66fca9a925f437fad62fb6d85014ce0299fccf303
2689367b205c16ce32ed4200942b8b8b1e262dfc70d9bc9fbc77c49699a4f1df
280220567060f2ca72ed20d4c286b40923b7d788dca5ef8335f1dbb3199c0845
2d3d0f180cbf7b9413572a8970b191797d2bf16aa350752580a3016b86d24a30
47f243e6834514b601b5447b84cb0fbbbed787b67aa56da2e8914e47613cb25b
4f72c7e466358b8de24d080be07815e7a0dcc8a21bee0da138a1747b4547837b
5097f2c6f4309f0407af1679b629fbc3b10f7afb295c6aa0c1e3bf087481c002
82edf26934430790b6b335b7cba525dc9bcc02485db979136ec2983da217408b
c2b43ecde6e726ba840a2aade0f4443d47050e6f1e6a3fb981448ae74f224b29
f8674bad9ffbede0a5b9a3452676b846865273f86f39aeb9df41ae43fe980952
fb567040abfba5705ef438233da6ca0df44fc81ceacffe1b5d8e1f3ffabc33fb