www.applause.com Open in urlscan Pro
18.245.86.18  Public Scan

Form analysis
0 forms found in the DOM

Text Content

Subscribe To Our Newsletter For The Latest In Digital Quality And Testing
 * Why Applause
    * The Applause Platform
      
      The comprehensive platform you need to deliver exceptional digital
      experiences
   
    * Crowdtesting
      
      Enabling digital quality and product excellence as the pioneer of
      crowdtesting
   
    * Our Solutions
      
      A wide range of testing solutions to cover all aspects of digital quality
   
    * Our Testing Community
      
      The world's largest community of digital experts – vetted, trained, fast,
      available
   
    * Industry Expertise
      
      Deep domain experience and expertise in the industries and use cases you
      need
   
    * Partnerships
      
      We work with partners to ensure our joint customers can release products
      with confidence
   
   Release Better, Faster
   
   See how Applause delivers authentic, real-world feedback on the quality of
   your digital experiences – so you can release with confidence.
   
   Watch Now
 * Our Solutions
    * Platform Features
    * Bring Your Own Testers
    * Test Case Management
   
    * Our Solutions
    * Manual Functional Testing
    * Automated Functional Testing
    * Integrated Functional Testing
    * In-Sprint Testing
    * User Experience Testing
    * Customer Journey Testing
    * Payment Testing
    * AI Training & Testing
    * Voice Testing
    * Accessibility Testing
    * Security Testing
   
    * by Industry
    * Retail
    * Media & Entertainment
    * Financial Services
    * Travel & Hospitality
    * Healthcare
   
    * by Job Function
    * Engineering Teams
    * Product Teams
    * QA Teams
   
    * by Technology
    * Website
    * Mobile App
    * Desktop App
    * Generative AI
    * Voice & AI
    * Metaverse
    * Amazon Alexa
    * Google Assistant
    * Automotive Tech
    * In-Store
    * IoT Device
    * AR & VR

 * About Us
    * About Applause
      
      Learn the what, where, why and brief history of Applause
   
    * Leadership Team
      
      Meet our executive and board team and check out their backgrounds
   
    * Life at Applause
      
      Get immersed in our culture, values, departments and perks
   
    * Newsroom
      
      Check out the latest press mentions and articles from Applause
   
    * Diversity, Equity & Inclusion
      
      Learn how DEI is ingrained in Applause and see our stats
   
   Join Our Team
   
   Are you passionate about helping the world’s most innovative brands deliver
   the best digital experiences? Find the right opportunity for you.
   
   Explore Job Listings
 * Resources
    * Blog
      
      Read the latest industry news on remote testing and digital quality
   
    * Resource Center
      
      Read and view our resources for the latest testing best practices
   
    * Product Sessions Live
      
      See our latest sessions focused on digital products and technology
   
    * Webinars
      
      Register today for our upcoming webinars on digital quality
   
    * Podcasts
      
      Listen to the latest episodes of the Ready, Test, Go. podcast
   
    * Customer Stories
      
      We help brands just like you launch uncommonly great digital assets with
      confidence
   
   State of Digital Quality, 2023
   
   Learn how to improve your digital quality processes and ensure exceptional
   customer experiences with special reports for your industry or region.
   
   Read Now
 * Get Started

Blog /  Dev & QA Trends /  Building Generative AI into Applause’s Workflows


BUILDING GENERATIVE AI INTO APPLAUSE’S WORKFLOWS

Like many other forward-looking organizations, Applause regularly evaluates ways
to capitalize on generative AI’s potential to boost productivity and streamline
business processes. To date, we have three production uses of genAI in place. In
this blog post, I’ll outline how we’re using generative AI for one of those use
cases, why we chose this specific application for the technology, how we tested
it, and what we learned through the process.

GENAI FOR TEST CASE MANAGEMENT

Test case management is a crucial practice for developing high-quality software.
By effectively managing your test cases, you can ensure that your software is
thoroughly tested and delivered with fewer defects. Well-written test cases take
less time to execute, which improves efficiency and throughput, ultimately
reducing costs. Conversely, poorly written ones can create confusion and bog
down testing efforts.

When we developed our test case management solution, we added the option to have
AI review test cases and recommend improvements. We’re using a double opt-in, so
each customer decides whether to enable that feature for their organization.
Once enabled someone on the QA team can request recommendations for a specific
test case.

Once a human sends a test case for review, OpenAI’s GPT suggests edits that
would improve the language, the clarity, or the steps in that test case. The
person who submitted the test case for review has the option to accept all of
the recommendations, accept a portion of the improvements, or pass and keep the
test case as-is.

WHY WE SELECTED THIS USE CASE

We identified test case recommendations as the easiest place for us to get
started with large language models. Test cases are very structured; they
typically don’t contain PII, screenshots, or random input from our testing
community or other people. They’re typically pre-production. We determined that
using genAI in this way was safe legally, and there's clear value that it can
provide.

HOW WE DEVELOPED AND TESTED THE APPLICATION

For the test case improvements, a lot of the work was integrating the feature
into our product in terms of a user interface that presents the test case to the
customer and lets them decide whether they like the changes or not. That was
done with a workflow implementation. The rest of the work, integrating with an
API, was a very simple thing.

Most of the time involved was focused on prompt engineering, which is an
interesting thing. It's kind of like programming — almost the test cases of the
future, to a degree — where you have to write a very structured set of
instructions for what you want the machine to do. For the large language model
to produce good output, you have to build a lot of instructions into the
prompts. For example, in our application, we had to specify that we wanted the
model to be precise and avoid adding humor. We told the model to use all of its
knowledge of the QA disciplines and take into account that we’re engineering
products in the mobile and web space, as opposed to manufacturing.

As far as testing, we did our normal QA through the community. We had a number
of our professional test case writers review the test cases before and after and
sign off on the recommendations from the model. Our bar was that at least 80% of
the time, the recommended test cases needed to be better than the original, so
that someone would accept the changes.

Long term, we’re collecting feedback on how many times people have asked to
clean up a test case and how many times the recommendations have been accepted.
Since this is a new feature, we need to let it run for a period of time, but
then we'll get that feedback. And we might get different answers from the
customers that are using the tool for feedback on their test cases versus our
own internal use. I would expect us to get less of an acceptance rate from our
community’s professional test case writers because we should be writing better
test cases to start, whereas customers’ internal teams may not be experts in
writing test cases. So maybe we get a higher acceptance rate there — that'll be
interesting data for us longer term.

LESSONS LEARNED

We spent weeks refining the prompt to get the results we wanted at this stage
and evolution of large language models. In the case where you're trying to do
repetitive things, like clean up hundreds of thousands of test cases, automating
has value. Refining that prompt makes sense, because your work one time to get a
good prompt will magnify itself over and over again.

We ran into some strange things, like one case where cleaning up a test case was
taking literally minutes for the machine to come back and provide us a
recommendation. We found that taking a single comma out of the prompt that we
had developed made it 60 times faster. When things aren't going the way you
want, be creative. There's no documentation out there that will tell you to do
something like that. You just have to keep adjusting things and then testing.
It’s experimentation – you want to make sure your prompt engineering is going to
yield a return on the investment of time you're putting into it.

In summary, we have a new tool in our bag of tricks as developers, as people, as
employees. As we're approaching problems today, whether it's in marketing or
engineering or finance, you have to ask not only if generative AI could help,
but whether it’s efficient. You'll definitely want to think about what you need
to share with that tool to get the answers you want. Consider PII and legal
compliance — you want to make sure that you're being safe with the data. You
shouldn’t give it to systems where you’re not sure how the AI is being trained
or the data is stored or retained. Even if a provider does claim they don't save
things, you're better off not trusting them at this stage.

You need to know what tools are out there, and then you do need to experience
the different tools. You need to spend time with generative AI to understand how
different systems work and the options available. Once you understand what one
is like, that goes into your bag of tricks and when you’re presented with a
problem to solve, you can turn to generative AI with a clearer sense of what
assistance you’ll get and where you have to be careful.




Webinars

TESTING GENERATIVE AI APPLICATIONS

Join Applause & Voicebot.ai to discuss the risks, challenges, and testing
strategies for safely deploying generative AI applications.

Watch now
Want to see more like this?
Dev & QA Trends
Share this:

Rob Mason
Chief Technology Officer
Published: December 18, 2023
Reading time: 9 min
Share this:

Dev & QA Trends


ADVICE FOR DIGITAL QUALITY INITIATIVES IN 2024 AND BEYOND


NEUTRALIZE THE EFFECTS OF DEFECTS

Dev & QA Trends


THE JOY OF AN ORGANIZED TESTING APPROACH


DUPLICATIVE EFFORTS AND REWORK ARE A RECIPE FOR FRUSTRATION

Dev & QA Trends


BUILDING GENERATIVE AI INTO APPLAUSE’S WORKFLOWS


WHILE APPLAUSE IS BEST KNOWN AS A TESTING COMMUNITY, WE’RE ALSO A SOFTWARE
DEVELOPMENT ORGANIZATION. LEARN HOW WE’RE BUILDING GENERATIVE AI INTO OUR
PLATFORM AND PROCESSES.

Dev & QA Trends


WORST DEFECT EVER: EXPERTS TELL THE TALE


SOFTWARE BUGS CAN BE MORE THAN A NUISANCE

Customer Stories & Insight


5 MOST COMMON PAYMENT TESTING QUESTIONS ANSWERED


EXPERT ANSWERS FROM OUR RECENT WEBINAR EXPLORING PAYMENT TESTING BEST PRACTICES

Dev & QA Trends


HOW TO INFUSE ADAPTABILITY INTO SOFTWARE TESTING


EMBRACING CHANGE HELPS BOTH THE INDIVIDUAL TESTER AND THE TEAM




GENERAL

 * Platform Login
 * Contact Us
 * Pricing Inquiry


RESOURCES

 * Blog
 * Newsroom
 * Resource Library
 * Webinars


ABOUT US

 * About Applause
 * Life at Applause
 * Join Our Team
 * Leadership Team


SOCIAL

 * Facebook
 * Instagram
 * LinkedIn

 * German
 * French
 * Japanese

English
© 2024 Applause App Quality, Inc.
APPLAUSE is a registered trademark of Applause App Quality, Inc.
Terms of Use Privacy Policy Accessibility Commitment Modern Slavery Act Do Not
Sell or Share My Personal Information Cookies Settings



By clicking “Accept All Cookies”, you agree to the storing of cookies on your
device to enhance site navigation, analyze site usage, and assist in our
marketing efforts.

Cookies Settings Reject All Accept All Cookies


Your Opt Out Preference Signal is Honored


PRIVACY PREFERENCE CENTER

When you visit any website, it may store or retrieve information on your
browser, mostly in the form of cookies. This information might be about you,
your preferences or your device and is mostly used to make the site work as you
expect it to. The information does not usually directly identify you, but it can
give you a more personalized web experience. Because we respect your right to
privacy, you can choose not to allow some types of cookies. Click on the
different category headings to find out more and change our default settings.
However, blocking some types of cookies may impact your experience of the site
and the services we are able to offer.
Allow All


MANAGE CONSENT PREFERENCES

STRICTLY NECESSARY COOKIES

Always Active

These cookies are necessary for the website to function and cannot be switched
off in our systems. They are usually only set in response to actions made by you
which amount to a request for services, such as setting your privacy
preferences, logging in or filling in forms. You can set your browser to block
or alert you about these cookies, but some parts of the site will not then work.
These cookies do not store any personally identifiable information.

PERFORMANCE COOKIES

Performance Cookies

These cookies allow us to count visits and traffic sources so we can measure and
improve the performance of our site. They help us to know which pages are the
most and least popular and see how visitors move around the site. All
information these cookies collect is aggregated and therefore anonymous. If you
do not allow these cookies we will not know when you have visited our site, and
will not be able to monitor its performance.

TARGETING COOKIES

Targeting Cookies

These cookies may be set through our site by our advertising partners. They may
be used by those companies to build a profile of your interests and show you
relevant adverts on other sites. They do not store directly personal
information, but are based on uniquely identifying your browser and internet
device. If you do not allow these cookies, you will experience less targeted
advertising.

Back Button


COOKIE LIST



Search Icon
Filter Icon

Clear
checkbox label label
Apply Cancel
Consent Leg.Interest
checkbox label label
checkbox label label
checkbox label label

Reject All Confirm My Choices