adamtheautomator.com Open in urlscan Pro
2606:4700:20::681a:64 Public Scan

Back to summary

URL:
https://adamtheautomator.com/invoke-webrequest/
Submission: On December 13 via manual (December 13th 2023, 3:19:17 am UTC) from JP — Scanned from JP

Form analysis
2 forms found in the DOM

GET https://adamtheautomator.com/

<form class="mt-3 mb-3 search xl:mt-0 lg:mb-0 xl:pr-2" method="get" action="https://adamtheautomator.com/">
  <label for="search-form-1">
    <span class="sr-only">Search for:</span>
    <input type="search" id="search-form-1" placeholder="Search …" class="min-w-[100px] xl:min-w-[200px] 2xl:min-w-[250px] w-full shadow appearance-none rounded py-2 px-2 text-slate-600 focus:outline-none" value="" name="s">
  </label>
  <input type="submit" class="sr-only" value="Search">
</form>

POST

<form method="post">
  <label class="mctb-label" for="mailchimp-top-bar__email">Don't be left behind with the ATA Learning Newsletter!</label>
  <input type="email" name="email" placeholder="Your email address" class="mctb-email" required="" id="mailchimp-top-bar__email">
  <input type="text" name="email_confirm" placeholder="Confirm your email" value="" autocomplete="off" tabindex="-1" class="mctb-email-confirm">
  <input type="submit" value="Subscribe Today!" class="mctb-button">
  <input type="hidden" name="_mctb" value="1">
  <input type="hidden" name="_mctb_timestamp" value="1701151813">
</form>

Text Content

ATA Learning
Tap to hide
ATA Learning
* Home
* Tutorials
* Guidebooks
* Instructors
* Get Paid to Write
* Advertising
* Recommended Resources
* Jobs

Search for:
*
*
*
*

POWERSHELL INVOKE-WEBREQUEST: A COMPREHENSIVE GUIDE

Published:16 June 2019 - 3 min. read

* PowerShell

Adam Bertram

Read more tutorials by Adam Bertram!

* Website>
* Twitter

Azure Cloud Labs: these FREE, on‑demand Azure Cloud Labs will get you into a
real‑world environment and account, walking you through step‑by‑step how to best
protect, secure, and recover Azure data.

Your Job!
Your Company!

$50,000 - $100,000

Get Started Today!

Table of Contents

* Basic Usage
* Downloading Files with Invoke-WebRequest
* Submitting a Form and Working with Sessions
* Resolving Short URIs
* Summary

XFacebookLinkedIn

Have you ever wanted to browse the web via the command line? Yea. Me neither.
But have you ever needed to pull information from a webpage, monitor a website
or submit information via automation? I have and I use Invoke-Webrequest to do
it!

The Invoke-WebRequest PowerShell cmdlet is the swiss army knife for the web.
This cmdlet can send any HTTP verb to a web service along with common things
like HTTP parameters, specify different HTTP headers and so on.
Invoke-WebRequest along with it’s brother, Invite-RestMethod are the two
PowerShell cmdlets you’ll want to familiarize yourself with if you need to do
any kind of web automation.

The Invoke-WebRequest cmdlet is a part of the Microsoft.PowerShell.Utility
module that comes with Windows PowerShell and PowerShell Core. This cmdlet was
included with PowerShell ever since v3 and it’s one that is extremely powerful
yet easy to use.

MY LATEST VIDEOS

Adam the Automator

0 seconds of 1 minute, 5 secondsVolume 0%

Press shift question mark to access a list of keyboard shortcuts
Keyboard ShortcutsEnabledDisabled
Play/PauseSPACE
Increase Volume↑
Decrease Volume↓
Seek Forward→
Seek Backward←
Captions On/Offc
Fullscreen/Exit Fullscreenf
Mute/Unmutem
Decrease Caption Size-
Increase Caption Size+ or =
Seek %0-9

Live
00:00
01:05
01:05

By using Invoke-WebRequest, PowerShell allows a developer to work with websites,
web services and REST APIs in a lot of different ways.

BASIC USAGE

At it’s most basic, the Invoke-WebRequest cmdlet sends an HTTP request method to
an endpoint such as a URI or URL. The cmdlet supports all of the common request
methods.

By far, the most common method is the GET method. This method reads information
such as information from a website or maybe querying a REST API. The method is
defined by using the Method parameter. Since we need an endpoint to query, we’ll
also need a URI as well. To keep this easy, I’ll pick any website. To
shamelessly promote TechSnips, I’ll choose techsnips.io.

Let’s say I want to get a listing of all of the latest published videos as shown
below.

Example webpage

I can get an HTML representation of this page by running Invoke-WebRequest -Uri
'https://techsnips.io' -Method GET. When I do this, Invoke-WebRequest downloads
the entire web page and returns an output with various parsed information around
the elements of the page.

Invoke-WebRequest response

To get the videos, I’ll need to do some digging. When I look at the links
property I see a commonality that all of the video links have a class of
ng-binding as shown below.

$result.Links | where {$_.class -eq ‘ng-binding’}

Once I know this, I can then find all of those elements and only return the
innerHTML property and voila!

$result.links | where {$_.class -eq ‘ng-binding’} | Select-Object innerHtml

DOWNLOADING FILES WITH INVOKE-WEBREQUEST

We can also use Invoke-WebRequest to download files from the web as well and
it’s really easy! We can download files by simply pointing Invoke-WebRequest at
a URI of a file and using the OutFile parameter to tell the cmdlet to save the
file to local disk.

As an example, below I’m downloading the SysInternals Handle utility and
expanding the zip file once downloaded. It’s really that easy!

Invoke-WebRequest -Uri 'https://download.sysinternals.com/files/Handle.zip' -OutFile C:\handle.zi Expand-Archive -Path C:\handle.zip

Copy

SUBMITTING A FORM AND WORKING WITH SESSIONS

We can use Invoke-WebRequest to also fill forms. To do this though, we commonly
need to work with web sessions. HTTP is a naturally stateless protocol and your
browser (in this case PowerShell) must be able to create a session which will be
used to track things like cookies, for example. A common form is a
login/password form so let’s login to a fictional website!

Let’s say our fictional login form is at the URL http://somewebsite.com. We’d
first need to run Invoke-WebRequest to download the HTML structure and create a
session.

$response = Invoke-WebRequest -Uri 'http://somewebsite.com' -SessionVariable rb

Copy

Once we do this, the response will have a Forms property we can then populate
with a username and password. In this case, the username is represented by a
field called user and the password should be in a field called password. This
will depend on the webpage.

$form = $response.Forms[0]
$form.Fields["user"] = "username"
$form.Fields["password"] = "password"

Copy

Once the form has been populated, we can then use Invoke-WebRequest again but
this time re-use the session we just created and automatically figure out the
URI to send it to by reading the Action property that’s on the form as shown
below.

$response = Invoke-WebRequest -Uri $form.Action -WebSession $rb -Method POST

Copy

If you’ve got all of the appropriate field names right and the webpage isn’t
doing any fancy, you should be logged in with the username and password inside
of the $rb web session. At this point, you can read various pages behind that
authentication if you use the $rb web session variable.

RESOLVING SHORT URIS

Finally, another great use of Invoke-WebRequest is resolving short URIs. Perhaps
you need to know what’s behind that shortened URL but don’t want to click on it
to find out! No problem. Using Invoke-WebRequest, we can read the AbsoluteUri
property from the parsed response it gives us!

Notice below I’m also using the UseBasicParsing parameter. By default,
Invoke-WebRequest tries to use Internet Explorer (IE) to parse the HTML
returned. This doesn’t work on systems without IE. To get around that, we can
use the UseBasicParsing parameter to still download the content but only lightly
parse it.

$Url = 'buff.ly/2sWvPOH'
$Web = Invoke-WebRequest -Uri $Url -UseBasicParsing
$Web.BaseResponse.ResponseUri.AbsoluteUri

Copy

SUMMARY

The Invoke-WebRequest cmdlet is one of the most versatile cmdlets that come with
PowerShell. If there’s an action that can be performed via a typical graphical
browser, the Invoke-WebRequest cmdlet can do it too. You can find an example of
using this cmdlet by taking a look at this article on monitoring REST APIs.

Hate ads? Want to support the writer? Get many of our tutorials packaged as an
ATA Guidebook.

Explore ATA Guidebooks

adamtheautomator.com Open in urlscan Pro 2606:4700:20::681a:64 Public Scan

Form analysis 2 forms found in the DOM

GET https://adamtheautomator.com/

POST

Text Content

adamtheautomator.com Open in urlscan Pro
2606:4700:20::681a:64 Public Scan

Form analysis
2 forms found in the DOM