blog.gurock.com Open in urlscan Pro
2606:4700:10::6816:314b Public Scan

Back to summary

Submitted URL:
https://info.gurock.com/e/831103/5-16-2022-utm-content-cli-tool/ymjn8/556084873?h=-E-DSOD0isaEl64yxEB5f3FRphfBAQyzwyFL6U...
Effective URL:
https://blog.gurock.com/announcing-the-testrail-cli-tool/?utm_source=pardot&utm_medium=email&utm_campaign=release&utm_te...
Submission: On May 17 via manual (May 17th 2022, 8:17:44 pm UTC) from IN — Scanned from DE

Form analysis
2 forms found in the DOM

GET https://www.gurock.com/testrail/search

<form role="search" method="get" id="searchform" class="search-form" action="https://www.gurock.com/testrail/search">
  <label class="screen-reader-text" for="q">Search for</label>
  <div class="search-input-wrapper">
    <input type="text" class="search-input" value="" name="q" id="q" placeholder="Search TestRail">
    <button class="search-button" id="searchsubmit">
      <i class="material-icons" id="search_icon">search</i>
    </button>
  </div>
</form>

GET https://blog.gurock.com/

<form role="search" method="get" id="searchform" class="searchform" action="https://blog.gurock.com/">
  <div>
    <label class="screen-reader-text" for="s">Search for:</label>
    <input type="text" value="" name="s" id="s">
    <input type="submit" id="searchsubmit" value="Search">
  </div>
</form>

Text Content

TestRail
Community
Search for
search
My Account account_circle
Blog

ANNOUNCING THE TESTRAIL CLI TOOL

by David Aaronson | May 16, 2022 | Uncategorized

The TestRail team is proud to announce the release of the TestRail CLI Tool.

Sending automated test results to TestRail just got a lot easier. By parsing and
uploading JUnit style test results into TestRail directly from the command line
or by running the CLI as part of an automated build pipeline, you can focus on
writing test code instead of needing to write, test, and maintain API calls in
every test script.

Centralized tracking of testing progress, measuring quality, and gaining
historical intelligence across both manual and automated testing is now faster
and easier than ever, bringing increased visibility and traceability to all
stakeholders in testing.
You can find the open source project on GitHub here

REPORT AUTOMATED TEST RESULTS FASTER WITH THE TESTRAIL CLI TOOL

The TestRail CLI Tool circumvents interfacing with the TestRail API to import
test results.

In order to implement TestRail automation features as part of an automated
testing solution, teams currently have to devise their own mechanism for
reporting the results of tests into TestRail by including TestRail API calls in
their automation frameworks. The TestRail CLI tool provides a programming
language agnostic mechanism that converts the most common test result formats
into a set of TestRail test results. Some of the most used test automation
frameworks that can easily generate reports which can be used by the TestRail
CLI tool include:

* JUnit
* Cypress.io
* Playwright
* Robot Framework
* Pytest
* NUnit

This tool paves the way for increased integration possibilities between
automated testing tools and TestRail, and makes the decision to invest in
TestRail as a test management solution even easier for teams heavily reliant
upon automated testing solutions.

The CLI tool also opens up new opportunities for tracking and comparing
historical results for ad hoc exploratory testing as well as security, load, and
performance testing, provided that the tools you are using to execute these
types of tests can generate JUnit style reports.

After reading through this article, you will be able to:

✔ Install Python 3.10 and pip3

✔ Install the TestRail CLI Tool

✔ Enable the TestRail API

✔ Set up the automation_id TestRail custom field

✔ Run the CLI Tool

✔ See your test results on TestRail

INSTALLING THE CLI TOOL

When you want to create or edit a webhook, you’ll go into the Edit Webhook view
in order to

Users can call the command line tool within the context of a CI pipeline or from
any computer’s command line interface. For example, once an automated test job
is completed, another call can be made to run the command line tool, thereby
pushing the results of their tests directly into TestRail.

It is also possible that the command line tool may be leveraged directly by
developers or testers running unit or other automated test suites locally as
well.
To install the CLI Tool, you just need to have Python 3 installed as well as
pip3 and run the following command:

$ pip3 install trcli

INSTALLING THE CLI TOOL

Before using the CLI tool, you first need to configure your TestRail instance as
follows:

1. Enable the TestRail API by accessing the Administration panel, going to Site
Settings, click on the API tab, and checking the Enable API option. You can
refer to the API Introduction page for more information.

0. Create a custom field in order to map your automated test cases code to the
actual TestRail cases. You can do so by accessing the Administration panel,
going to Customizations on the right side and clicking Add Field. After
you’ve reached the field creation screen, as per the image below, there are
two requirements for this custom field:
1. The System Name must be automation_id
2. The Type must be String.

IMPORTING YOUR JUNIT TEST RESULTS TO TESTRAIL

The CLI Tool is designed to be simple and efficient. Once it has been installed
and your TestRail instance is properly configured, a JUnit results file can be
passed through the command line to quickly create a run and add testing results.
By way of an example, we set up an example TestRail project to accept incoming
API requests, and pushed the sample report below to our TestRail instance via
command line. Notice the -y option on the command itself, which skips the
prompts to automatically create new tests. This is useful when running the CLI
tool through your CI tools.

Note: It is recommended that you create an API key under My Settings on your
account and use it instead of your password. This will prevent you from exposing
your password and thereby creating a security risk.

<testsuites name="test suites root">
<testsuite failures="0" errors="0" skipped="1" tests="1" time="0.05" name="tests.LoginTests">
<properties>
<property name="setting1" value="True"/>
</properties>
<testcase classname="tests.LoginTests" name="test_case_1" time="159">
<skipped type="pytest.skip" message="Please skip">
skipped by user
</skipped>
</testcase>
<testcase classname="tests.LoginTests" name="test_case_2" time="650">
</testcase>
<testcase classname="tests.LoginTests" name="test_case_3" time="159">
<failure type="pytest.failure" message="Fail due to...">
failed due to…
</failure>
</testcase>
</testsuite>
</testsuites>

$ trcli -y \
> -h https://trcli.testrail.io \
> --project "TRCLI Test 1" \
> --username trcli.user@gmail.com \
> --password TestRail123! \
> parse_junit \
> --title "Automated Tests Run" \
> -f results.xml

Checking project. Done.
Adding missing sections to the suite.
Found test cases not matching any TestRail case (count: 3)
Adding missing test cases to the suite.
Adding test cases: 3/3, Done.
Creating test run. Done.
Adding results: 3/3, Done.
Submitted 3 test results in 6.5 secs.

Once the import process is complete, you will see a new run titled Automated
Tests Run within the TRCLI Test 1 project:

By opening the test run, you can see the results for each test case. You can
then drill further into a failed test case and check the error message that was
imported directly from the JUnit report. This can be helpful to have a quick
overview of what went wrong during the test.

Notice that the test cases on this test run did not previously exist on
TestRail. These test cases were created by the CLI tool, which uses the pattern
classname.name (attributes from the JUnit report) to fill in the Automation ID
field so that it can import the results to the correct case on TestRail on
future test runs, assuming the test name and classname don’t change. On the
image below, you can see that test_case_3 has the Automation ID value filled in
with the value tests.LoginTests.test_case_3, as expected, so that the CLI tool
will be able to identify it in future test result imports.

OTHER USEFUL FEATURES

USING CONFIG FILES TO STORE ALTERNATE CONFIGURATIONS

Quickly and easily submit results to different instances or projects, use
different credentials, or other preset parameters in your commands by using an
alternate config file. The configuration file will be written in YAML format,
named config.yml, and stored in the same directory as the TRCLI executable file,
unless otherwise specified. Environment variables can also be used. If a
configuration file is referenced in the command, all parameters within the
configuration file will override the environment variables. Any parameters
specified in the command will override the config file.

The following example displays the use of an alternate configuration file that
stores users credentials:

host: https://trcli.testrail.io
project: TRCLI Test 1
username trcli.user@gmail.com
password TestRail123!
project: TRCLI Test 1
title: Automated Tests Run

$ trcli -y \
> --config alternate_config.yaml \
> parse_junit \
> -f results.xml

UPDATING TEST RESULTS

Imagine you already ran the CLI tool and you have your automated test results on
TestRail, but some of the tests failed and you want to rerun those and update
the existing test run with the new results. In order to do so, you just need to
pass the –run-id argument and the CLI tool will update the test run with that
id.

Checking project. Done.
Adding results: 3/3, Done.
Closing test run. Done.
Submitted 2 test results in 2.5 secs.

You should now see the new test result on the test details panel. This is one
way to keep track of your automated test results under the same test run.

CLOSING THE TEST RUN

If you want to immediately close your newly created test run, you can simply
pass the –close-run argument and the CLI tool will perform that action after all
the results have been added. This is useful if you don’t want to allow changes
to be made to the results after the run has finished.

Checking project. Done.
Creating test run. Done.
Adding results: 3/3, Done.
Closing test run. Done.
Submitted 3 test results in 3.2 secs.

You can find your test run under the Completed test runs section.

ADDED BENEFITS

MULTITHREADING

The CLI Tool was made to quickly upload test results through a series of API
calls, which are inherently impacted by network latency. To optimize the
performance, multiple threads are used to execute a series of API calls at once.

RETRIES & THROTTLING

While multithreading allows the CLI Tool to make calls to the TestRail API more
efficiently, it can also be troublesome if it hits API rate limits. To
circumvent this, retry and throttling mechanisms were put in place. If the rate
limit is hit, the CLI Tool will wait the appropriate amount of time and retry
the call for a successful response.

UNIT TESTED

To make it easier for contributors to make changes to the open source TestRail
CLI project with confidence, we included a vast number of unit tests to avoid
unwanted side effects.

EXTENSIBLE

Need a parser for a different type of report? Simply create a new parser class,
implement it according to the CLI Tool data model and you can trust the
remaining pieces of the code to upload your test results! You can also implement
new functionalities without having to fiddle with existing code.

Search for:

TAGS

* Agile (127)
* Announcement (33)
* Automation (16)
* Books (2)
* Business (43)
* Continuous Delivery (29)
* General (424)
* Gurock Software (188)
* Interviews (4)
* Performance (17)
* Programming (68)
* Security (15)
* SmartInspect (136)
* Software Quality (200)
* TestRail (136)
* Uncategorized (23)
* Virtualization (12)
* Webinar (22)
* Websites (24)

Contact
Security
Privacy Policy
Terms

We use cookies on our website to give you the most relevant experience by
remembering your preferences and repeat visits. By clicking “Accept”, you
consent to the use of ALL the cookies.

Do not sell my personal information.
Cookie SettingsAccept
Manage consent
Close

PRIVACY OVERVIEW

This website uses cookies to improve your experience while you navigate through
the website. Out of these, the cookies that are categorized as necessary are
stored on your browser as they are essential for the working of basic
functionalities of the ...
Necessary
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly.
These cookies ensure basic functionalities and security features of the website,
anonymously.

CookieDurationDescriptioncookielawinfo-checkbox-analytics11 monthsThis cookie is
set by GDPR Cookie Consent plugin. The cookie is used to store the user consent
for the cookies in the category "Analytics".cookielawinfo-checkbox-functional11
monthsThe cookie is set by GDPR cookie consent to record the user consent for
the cookies in the category "Functional".cookielawinfo-checkbox-necessary11
monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to
store the user consent for the cookies in the category
"Necessary".cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR
Cookie Consent plugin. The cookie is used to store the user consent for the
cookies in the category "Other.cookielawinfo-checkbox-performance11 monthsThis
cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the
user consent for the cookies in the category
"Performance".viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie
Consent plugin and is used to store whether or not user has consented to the use
of cookies. It does not store any personal data.

Functional
Functional
Functional cookies help to perform certain functionalities like sharing the
content of the website on social media platforms, collect feedbacks, and other
third-party features.
Performance
Performance
Performance cookies are used to understand and analyze the key performance
indexes of the website which helps in delivering a better user experience for
the visitors.
Analytics
Analytics
Analytical cookies are used to understand how visitors interact with the
website. These cookies help provide information on metrics the number of
visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and
marketing campaigns. These cookies track visitors across websites and collect
information to provide customized ads.
Others
Others
Other uncategorized cookies are those that are being analyzed and have not been
classified into a category as yet.
SAVE & ACCEPT

blog.gurock.com Open in urlscan Pro 2606:4700:10::6816:314b Public Scan

Form analysis 2 forms found in the DOM

GET https://www.gurock.com/testrail/search

GET https://blog.gurock.com/

Text Content

blog.gurock.com Open in urlscan Pro
2606:4700:10::6816:314b Public Scan

Form analysis
2 forms found in the DOM