What’s the Quality of Your Quality Assurance Program?

You hired a great software development team. They’re smart, fast, and have developed products that should leave the competition in the dust. But, lately, you’ve noticed a nagging trend. Revisions and bug fixes are starting to pop up with greater frequency. Projects are running behind schedule and over budget. Now, every time you hit the release button you say a silent prayer that the last code updates have not injected new defects.

Eventually, the original excitement that came as a result of being perceived as a market leader begins to erode, replaced by bad customer reviews and a rapidly demoralized team. 

Your Problem? Lack of Quality Assurance (QA)

QA is all too often treated like a red-headed stepchild when, in reality, it’s one of the most important steps in the software development process. It’s also one of the most misunderstood. Here are some of the more common QA issues that, if left unresolved can have devastating effects on your bottom line. 

The Wrong People May Be “Testing”

There is a misconception that anyone can test. It’s often delegated to developers, business analysts, product owners, stakeholders, and even end-users. Investing a lot of money into a product but not investing in the process to ensure its quality doesn’t make much sense. 

No matter how talented and experienced programmers are, it’s simply not possible for them to test their own logic without encountering “implementation bias.” The goal of a developer should be focusing on innovation, creativity, and writing code to solve user objectives. It’s not to run endless test scenarios. 

Enter the Software Quality Assurance Analyst

In contrast, the job of a dedicated Quality Assurance Analyst is to verify, validate, and explore the system to predict and prevent technical risks. Armed with that analysis, stakeholders can then make informed decisions. Quality Assurance Analysts conduct static requirements analysis, functional, integration, system, end-to-end, performance, stress, load, compatibility, and exploratory testing at all layers and environments of the software product. Analysts create testing documentation and detailed bug reports to ensure not a single defect is left unaddressed. 

The QA Practice is Not Fully Integrated into the Development Process

If your team is using the classic Waterfall methodology—where testing follows when code is fully written—then you’re creating an environment where QA is at an inherent disadvantage. Consequently, your time to market will increase as defects that could have been prevented in the earliest stages of the software development life cycle (SDLC) are left unidentified. And all too often, when a project is “completed” and testing is the bottleneck, code is pushed prematurely as business pressures mount to launch.

QA & Agile—A Match Made in Software Heaven

Adopting the Agile methodology brings more flexibility to the development process best captured by the mantra of “Test early, test often, test together.” Rather than viewed as adversarial elements, Agile brings Development and QA together working in close collaboration uniting team members around a common goal: create and deliver high-quality products on time and within budget. The role of QA is significantly advanced in an Agile team. Within Agile, testing is not considered a step in the process, it is the essence of the process. It should be built into the product from day one!

It’s Not Easy to Hire the Right Quality Assurance Analyst

While a Quality Assurance Analyst needs to possess such classic skill sets as test planning and execution, defect management, the role of a QA Analyst has changed. Equally important are:

  • Critical  thinking, mental agility, and creativity
  • Understanding end-user needs help deliver business value, improve software quality, and increase delivery velocity. 

Today, being a QA Analyst means being a quality advocate who spreads the culture of quality across development teams using the language of programmers. 

Automate Your Testing 

Regression test cycle has grown to the point that manual testing takes too much time and a regression test suite is hard to maintain. This is why you should automate routine testing. By doing so, you’ll be able to quickly provide development teams with critical feedback, reduce time to market, improve product quality, and reduce overall costs. Automated testing provides you with the luxury of running enough exploratory, usability, and acceptance testing against the latest builds, searching for unusual or unexpected behavior of the system. 

There are multiple approaches and tools, both open source and paid that you can use to automate your testing. But, it’s worth noting that choosing a tool is just the beginning of the process. You also need the right strategy and the right resources to administer the process. 

Some Questions You Should Be Asking Yourself

  • Are you satisfied with the quality of your software?
  • Does your company keep investing in technology development processes, but you still struggle with production defects?
  • Do your software engineers spend their time on testing rather than developing new features?
  • Does your HR/Recruiting department have expertise in hiring the right Quality Analyst for your company?
  • Is there a perception that testing is a bottleneck?
  • Would you like to scale your QA team?

If any of the above questions resonate with you, let’s talk!

PeakActivity can help raise the quality of your software product and software development process, achieving superior product quality standards while providing for an exceptional user experience. After an in-depth analysis of the product, requirements, development process, and stakeholders’ priorities our experts will:

  • Tailor a team of qualified professionals that perfectly fit your company’s needs and corporate culture.
  • Seamlessly embed our quality practice into the existing software development life cycle, making sure the development team feels comfortable and supported.
  • Develop a testing strategy that provides fast, efficient, and cost-effective end-to-end solutions. 

As a part of a development team, our Quality Engineers will:

  • Create company quality standards and guidelines.
  • Conduct all types of testing activities including usability, functional, integration, regression, performance, test automation, A/B testing, and more.
  • Provide static analysis of business requirements to identify unclear or missed requirements and prevent rework
  • Create and execute a regression testing suite, ensuring that the system is reliable and stable.
  • Provide automation of the testing process, utilizing up-to-date industry approaches and tools. 
  • Create test documentation, recordings of testing coverage, and test results.

Are you interested in elevating the quality of your Quality Assurance?

Fill out the form below or visit our Contact Us page.

Read More

Power Analysis & Sample Size Estimation

The Experiment

Performing power analysis and sample size estimation is an important aspect of experimental design. Without these calculations, the sample size may be too high or too low. If the sample size is too low, the experiment will lack the precision to provide reliable answers to the questions it is investigating. In this case, it would be wise to alter or abandon the experiment. If the sample size is too large, time and resources will be wasted, often for minimal gain.

How - Gather

For each test, we will gather the following four data points before any code or resources are used. What is the primary performance indicator (KPI)? By how much do we believe our hypothesis will effect that KPI (effect size)? Our standard can be 1%, 2% ,3% What is the acceptable minimum confidence level – 90%, 95%, 99% (significance level)? What is the acceptable minimum power level- 70%, 80%, 90%? Often considered to be between .80 and. 90. Think of “Power” as the strength of the experiment. Statistical power is the probability that the test will detect an effect that actually exists. What is the current traffic size on the page being tested?

Why

Calculate

With these data points, (effect size, sample size, significance level, power) we can enter three of the four quantities and the fourth is calculated. The basic idea of calculating power or sample size is to leave out the argument that you want to calculate. If you want to calculate power, then leave the power argument out of the equation. If you want to calculate sample size, leave ‘n’ out of the equation. Whatever parameter you want to calculate is determined from the others.

What

Power Analysis - Checkout

Hypothesis/Success Criteria: If we clearly call out the guest checkout option then we will increase conversion by at least 2%.

What is the optimal sample size for the given hypothesis?

Sample Size (n) = Unknown?
Effect Size (d) = 2%
Power = 80%
Sig Level (alpha/confidence level) = 0.05 or 95%
Sample Size = 19,625

This tells us that we need ~20k sessions to reach 95% confidence to see a 2% increase in conversion at an 80% probability that the detected lift actually exists. If we do not reach a 2% increase in conversion at 95% conf. in the optimal sample size then we failed to reject the null hypothesis.

If we met the 2% increase in conversion rate at 95% confidence in ~20k then we would have rejected the null hypothesis.

Power Analysis For - Checkout - Guest Checkout - Current Results

What is the Power of our current test results? Sample Size (n) = 30,946 Effect Size (d) = 1.8% Power = Unknown? Sig Level (alpha/confidence level) = 0.12 or 88% Power = ~90% This tells us that there is a 90% probability our test we will be able to detect a change. However, there is only an 88% confidence level in that change. What do we do? We could accept 88% as “good enough”. We could re-run our power analysis with a smaller effect size. This will increase the sample size needed. Continue running the test.

Get Your FREE A/B Assessment Now!

Sign me up for get a FREE Assestment*
Read More

A/B Testing Results & Examples

Cart Page Checkout Call To Actions

The Experiment

Defining “Low-Touch” Commerce

Short Description: In the cart, the main call to action is the checkout button. The current style has many competing calls to action without a visual hierarchy, cluttering the design and in some cases, confusing customers.

Hypothesis/Success Criteria: If we re-design the secondary call to actions on the cart page with different colors and sizes then we will increase the number of checkout button clicks and therefore increasing cart conversion.

 

 

CONTROL
EXPERIMENT

The Details:

How Does It Work?
Utilizing our partner Curalate that lets you use social content and audiences to sell more effectively online. They allow us to bring social content onto our sites that inspire visitors and encourages purchases. The location of the social content displayed on site is important to the customer’s journey.
Why are we testing this?
Integrate images and videos from customers, influencers, partners, and our own social media accounts throughout your site to help people envision your products in their lives. Especially in the case of furniture, where design, trending styles, comfort, size, and fit are all rather difficult to display through a website.

The Results

Below is the recap of the hypothesis, success criteria, along with our go-forward recommendation the Homepage – Curalate Carousel Placement. Hypothesis/Success Criteria: If we move the Curalate carousel below the main homepage slider then we will increase customer engagement and therefore decrease the bounce rate. Recommendation: Based on our hypotheses, success metrics, and sample size we can say that our hypothesis was correct. We see a significant decrease in the home page bounce rate when the user-generated content is below the homepage sliders. This is leading to a significant increase in the homepage click-through rate. We see no negative effect on any primary revenue KPI’s. Our recommendation is to move the winning variant to 100%.

Want to learn more about how social influencers
can amplify your brand?

Reach out to our PeakActivity Social Squad for a free audit.

Sign me up for a FREE audit of my social media strategy.*
Read More

A/B Test: Home Page User-Generated Content

A/B Testing Results & Examples

Display User-Generated Content On Home Page

The Experiment

Short Description: Move the social media user generated carousel (UGC) below the main homepage sliders.

Hypothesis/Success Criteria: If we move the UGC carousel below the main homepage slider then we will increase customer engagement and therefore decrease bounce rate.

Supporting Data: User-generated content has a big influence on purchasing decisions.
According to Gartner’s research, 84% of millennials are likely to be influenced to make a purchase based upon user-generated content that is created by strangers

CONTROL

EXPERIEMENT



The Details

How Does It Work?

Utilizing our partner Curalate that lets you use social content and audiences to sell more effectively online. They allow us to bring social content onto our sites that inspire visitors and encourages purchases. The location of the social content displayed on site is important to the customer’s journey.

Why Are We Testing This?

Integrate images and videos from customers, influencers, partners, and our own social media accounts throughout your site to help people envision your products in their lives. Especially in the case of furniture, where design, trending styles, comfort, size, and fit are all rather difficult to display through a website.



The Results

Below is the recap of the hypothesis, success criteria, along with our go-forward recommendation the Homepage – Curalate Carousel Placement.

Hypothesis/Success Criteria: If we move the Curalate carousel below the main homepage slider then we will increase customer engagement and therefore decrease the bounce rate.

Recommendation: Based on our hypotheses, success metrics, and sample size we can say that our hypothesis was correct. We see a significant decrease in home page bounce rate when the user-generated content is below the homepage sliders. This is leading to a significant increase in the homepage click-through rate. We see no negative effect on any primary revenue KPI’s.  Our recommendation is to move the winning variant to 100%

DeviceBounce Rate DecreaseHome page Click through Rate (CTR) LiftProduct Detail Page View LiftTotal SessionsWin / Lost
All Devices-3.1% (91% Conf.)0.38% (93% Conf.)0.6% (57% Conf.)113,059Win
Read More