blog.gurock.com
Open in
urlscan Pro
2606:4700:10::6816:304b
Public Scan
URL:
https://blog.gurock.com/announcing-the-testrail-cli-tool/?utm_source=pardot&utm_medium=email&utm_campaign=release&utm_te...
Submission: On May 17 via manual from IN — Scanned from DE
Submission: On May 17 via manual from IN — Scanned from DE
Form analysis
2 forms found in the DOMGET https://www.gurock.com/testrail/search
<form role="search" method="get" id="searchform" class="search-form" action="https://www.gurock.com/testrail/search">
<label class="screen-reader-text" for="q">Search for</label>
<div class="search-input-wrapper">
<input type="text" class="search-input" value="" name="q" id="q" placeholder="Search TestRail">
<button class="search-button" id="searchsubmit">
<i class="material-icons" id="search_icon">search</i>
</button>
</div>
</form>
GET https://blog.gurock.com/
<form role="search" method="get" id="searchform" class="searchform" action="https://blog.gurock.com/">
<div>
<label class="screen-reader-text" for="s">Search for:</label>
<input type="text" value="" name="s" id="s">
<input type="submit" id="searchsubmit" value="Search">
</div>
</form>
Text Content
TestRail Community Search for search My Account account_circle Blog ANNOUNCING THE TESTRAIL CLI TOOL by David Aaronson | May 16, 2022 | Uncategorized The TestRail team is proud to announce the release of the TestRail CLI Tool. Sending automated test results to TestRail just got a lot easier. By parsing and uploading JUnit style test results into TestRail directly from the command line or by running the CLI as part of an automated build pipeline, you can focus on writing test code instead of needing to write, test, and maintain API calls in every test script. Centralized tracking of testing progress, measuring quality, and gaining historical intelligence across both manual and automated testing is now faster and easier than ever, bringing increased visibility and traceability to all stakeholders in testing. You can find the open source project on GitHub here REPORT AUTOMATED TEST RESULTS FASTER WITH THE TESTRAIL CLI TOOL The TestRail CLI Tool circumvents interfacing with the TestRail API to import test results. In order to implement TestRail automation features as part of an automated testing solution, teams currently have to devise their own mechanism for reporting the results of tests into TestRail by including TestRail API calls in their automation frameworks. The TestRail CLI tool provides a programming language agnostic mechanism that converts the most common test result formats into a set of TestRail test results. Some of the most used test automation frameworks that can easily generate reports which can be used by the TestRail CLI tool include: * JUnit * Cypress.io * Playwright * Robot Framework * Pytest * NUnit This tool paves the way for increased integration possibilities between automated testing tools and TestRail, and makes the decision to invest in TestRail as a test management solution even easier for teams heavily reliant upon automated testing solutions. The CLI tool also opens up new opportunities for tracking and comparing historical results for ad hoc exploratory testing as well as security, load, and performance testing, provided that the tools you are using to execute these types of tests can generate JUnit style reports. After reading through this article, you will be able to: ✔ Install Python 3.10 and pip3 ✔ Install the TestRail CLI Tool ✔ Enable the TestRail API ✔ Set up the automation_id TestRail custom field ✔ Run the CLI Tool ✔ See your test results on TestRail INSTALLING THE CLI TOOL When you want to create or edit a webhook, you’ll go into the Edit Webhook view in order to Users can call the command line tool within the context of a CI pipeline or from any computer’s command line interface. For example, once an automated test job is completed, another call can be made to run the command line tool, thereby pushing the results of their tests directly into TestRail. It is also possible that the command line tool may be leveraged directly by developers or testers running unit or other automated test suites locally as well. To install the CLI Tool, you just need to have Python 3 installed as well as pip3 and run the following command: $ pip3 install trcli INSTALLING THE CLI TOOL Before using the CLI tool, you first need to configure your TestRail instance as follows: 1. Enable the TestRail API by accessing the Administration panel, going to Site Settings, click on the API tab, and checking the Enable API option. You can refer to the API Introduction page for more information. 0. Create a custom field in order to map your automated test cases code to the actual TestRail cases. You can do so by accessing the Administration panel, going to Customizations on the right side and clicking Add Field. After you’ve reached the field creation screen, as per the image below, there are two requirements for this custom field: 1. The System Name must be automation_id 2. The Type must be String. IMPORTING YOUR JUNIT TEST RESULTS TO TESTRAIL The CLI Tool is designed to be simple and efficient. Once it has been installed and your TestRail instance is properly configured, a JUnit results file can be passed through the command line to quickly create a run and add testing results. By way of an example, we set up an example TestRail project to accept incoming API requests, and pushed the sample report below to our TestRail instance via command line. Notice the -y option on the command itself, which skips the prompts to automatically create new tests. This is useful when running the CLI tool through your CI tools. Note: It is recommended that you create an API key under My Settings on your account and use it instead of your password. This will prevent you from exposing your password and thereby creating a security risk. <testsuites name="test suites root"> <testsuite failures="0" errors="0" skipped="1" tests="1" time="0.05" name="tests.LoginTests"> <properties> <property name="setting1" value="True"/> </properties> <testcase classname="tests.LoginTests" name="test_case_1" time="159"> <skipped type="pytest.skip" message="Please skip"> skipped by user </skipped> </testcase> <testcase classname="tests.LoginTests" name="test_case_2" time="650"> </testcase> <testcase classname="tests.LoginTests" name="test_case_3" time="159"> <failure type="pytest.failure" message="Fail due to..."> failed due to… </failure> </testcase> </testsuite> </testsuites> $ trcli -y \ > -h https://trcli.testrail.io \ > --project "TRCLI Test 1" \ > --username trcli.user@gmail.com \ > --password TestRail123! \ > parse_junit \ > --title "Automated Tests Run" \ > -f results.xml Checking project. Done. Adding missing sections to the suite. Found test cases not matching any TestRail case (count: 3) Adding missing test cases to the suite. Adding test cases: 3/3, Done. Creating test run. Done. Adding results: 3/3, Done. Submitted 3 test results in 6.5 secs. Once the import process is complete, you will see a new run titled Automated Tests Run within the TRCLI Test 1 project: By opening the test run, you can see the results for each test case. You can then drill further into a failed test case and check the error message that was imported directly from the JUnit report. This can be helpful to have a quick overview of what went wrong during the test. Notice that the test cases on this test run did not previously exist on TestRail. These test cases were created by the CLI tool, which uses the pattern classname.name (attributes from the JUnit report) to fill in the Automation ID field so that it can import the results to the correct case on TestRail on future test runs, assuming the test name and classname don’t change. On the image below, you can see that test_case_3 has the Automation ID value filled in with the value tests.LoginTests.test_case_3, as expected, so that the CLI tool will be able to identify it in future test result imports. OTHER USEFUL FEATURES USING CONFIG FILES TO STORE ALTERNATE CONFIGURATIONS Quickly and easily submit results to different instances or projects, use different credentials, or other preset parameters in your commands by using an alternate config file. The configuration file will be written in YAML format, named config.yml, and stored in the same directory as the TRCLI executable file, unless otherwise specified. Environment variables can also be used. If a configuration file is referenced in the command, all parameters within the configuration file will override the environment variables. Any parameters specified in the command will override the config file. The following example displays the use of an alternate configuration file that stores users credentials: host: https://trcli.testrail.io project: TRCLI Test 1 username trcli.user@gmail.com password TestRail123! project: TRCLI Test 1 title: Automated Tests Run $ trcli -y \ > --config alternate_config.yaml \ > parse_junit \ > -f results.xml UPDATING TEST RESULTS Imagine you already ran the CLI tool and you have your automated test results on TestRail, but some of the tests failed and you want to rerun those and update the existing test run with the new results. In order to do so, you just need to pass the –run-id argument and the CLI tool will update the test run with that id. $ trcli -y \ > -h https://trcli.testrail.io \ > --project "TRCLI Test 1" \ > --username trcli.user@gmail.com \ > --password TestRail123! \ > parse_junit \ > --title "Automated Tests Run" \ > --run-id 32 > -f results.xml Checking project. Done. Adding results: 3/3, Done. Closing test run. Done. Submitted 2 test results in 2.5 secs. You should now see the new test result on the test details panel. This is one way to keep track of your automated test results under the same test run. CLOSING THE TEST RUN If you want to immediately close your newly created test run, you can simply pass the –close-run argument and the CLI tool will perform that action after all the results have been added. This is useful if you don’t want to allow changes to be made to the results after the run has finished. $ trcli -y \ > -h https://trcli.testrail.io \ > --project "TRCLI Test 1" \ > --username trcli.user@gmail.com \ > --password TestRail123! \ > parse_junit \ > --title "Automated Tests Run" \ > --close-run true > -f results.xml Checking project. Done. Creating test run. Done. Adding results: 3/3, Done. Closing test run. Done. Submitted 3 test results in 3.2 secs. You can find your test run under the Completed test runs section. ADDED BENEFITS MULTITHREADING The CLI Tool was made to quickly upload test results through a series of API calls, which are inherently impacted by network latency. To optimize the performance, multiple threads are used to execute a series of API calls at once. RETRIES & THROTTLING While multithreading allows the CLI Tool to make calls to the TestRail API more efficiently, it can also be troublesome if it hits API rate limits. To circumvent this, retry and throttling mechanisms were put in place. If the rate limit is hit, the CLI Tool will wait the appropriate amount of time and retry the call for a successful response. UNIT TESTED To make it easier for contributors to make changes to the open source TestRail CLI project with confidence, we included a vast number of unit tests to avoid unwanted side effects. EXTENSIBLE Need a parser for a different type of report? Simply create a new parser class, implement it according to the CLI Tool data model and you can trust the remaining pieces of the code to upload your test results! You can also implement new functionalities without having to fiddle with existing code. Search for: RECENT POSTS * How to Integrate Test Management in Your GitLab CI Pipelines * Announcing the TestRail CLI Tool * What is the Role of QA in DevOps? * How to Manage and Track Exploratory Testing * TestRail 7.5 Cloud & Server updates with Webhooks, new SSO implementations, & MFA TAGS * Agile (127) * Announcement (33) * Automation (16) * Books (2) * Business (43) * Continuous Delivery (29) * General (424) * Gurock Software (188) * Interviews (4) * Performance (17) * Programming (68) * Security (15) * SmartInspect (136) * Software Quality (200) * TestRail (136) * Uncategorized (23) * Virtualization (12) * Webinar (22) * Websites (24) Contact Security Privacy Policy Terms © 2003-2022 Gurock Software GmbH We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies. Do not sell my personal information. Cookie SettingsAccept Manage consent Close PRIVACY OVERVIEW This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the ... Necessary Necessary Always Enabled Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously. CookieDurationDescriptioncookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data. Functional Functional Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features. Performance Performance Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors. Analytics Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc. Advertisement Advertisement Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads. Others Others Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet. SAVE & ACCEPT