Home Blog Page 11

How to Write an effective bug report?

0

Introduction:

A Software Bug can be described through the detailed explanation of the unexpected behavior happening as a result of some action performed on the software application. In bug reports, try to make very clear what are actual facts and what are speculations. Leave out speculations if you want to, but don’t leave out facts.

An effective bug report can save time for developers and testers and other stakeholders. Ideally, a ticket should stand alone, including all the information needed to reproduce the bug and it should be written in a way that even a non-technical person can understand and reproduce it.

 

First thing first: Try to produce the bug at least 3 times

Yes, 3 times at least? You ask why? Believe me, you don’t want to take 20 to 30 minutes of your time to create a bug report and then see it’s not there anymore! How could it possible? that’s easy, a part of a whole system (including hardware, software, and network) might fail that specific time and after it works again correctly; so what do you want to do? Begging the system for that bug back?!

 

Report one thing per case because minimalist is beautiful!

Never include more than one bug or failure in a case. This can make everyone confuse and it’s against clarity. Keep it minimal and simple and make everyone happy!!

 

It’s not personal, keep it professional:

Some bug reports look like friendly chats. They started with some phrases such as: “Joe, as we discussed before” or “As Jane notified us that day”. You, yourself will not remember what discussion or which meeting you were mentioned there, so you cannot expect the others sometime at future can understand that. If you need to mention such conversation, bring a brief summary of that to save everyone from confusion.

 

Bug Report Standards

Here I bring some principles and suggestions to write a standard and effective bug report. You may agree or disagree with me on this; I shall be grateful if I have your comments here (find it at the end of article).

 

Title:

The title should indicate the main problem and includes basic information about the place it points. It needs to be highly readable and easily referable. If you will use acronyms on the case description, use complete words on the title and add the short in parenthesis.

Since the case title could always be viewed inside of other cases, I suggest adding the related case number at the end of title as a reference.

Example: Wrong Credit Card (CC) number is accepted by payment system (ref. #2034)

 

Summary:

The summary should stand alone to let the reader understand and identify the issue quickly. The goal of the summary is to make the report searchable and uniquely identifiable. Bug summary does not include technical details and it is just bringing a brief explanation on this in one paragraph.

In short, bug summary should be written in a way that even the grandma next door can understand it!

Example: In our payment system where the users are asked for their Credit Card number, the system will accept whatever numbers they put and so it will be considered as successful payment where it is not.

 

Description

In a good bug description, you describe the issue precisely and completely in both general and technical words, statements and expressions. You may include all the necessary and relative information.

Please note: Only relevant information should be included and if in case you need to bring other relative cases for this case, indicate case numbers as a reference and where the information is located. If the bug only appears after a certain sequence of events, then at the next step you will bring those list of events.

Also, you can bring some references in your description if you think they are necessary to read by stakeholders.

 

Example: Bank Identification Numbers (BINs), which are the first six-digits of the account number, are fundamental to payments. They identify the issuing institution for the account and ensure that each transaction is routed correctly. (1)

On July 2017, MasterCard announced that they will have a new (Series 2) the initial number of their new clients. Previously this company used Series 5.

Currently, our system is using Luhn algorithm to validate the identification of CC numbers. (2)

We need to change our code in a way that our system accepts new card numbers by MC.

References:

  • https://www.mastercard.us/en-us/issuers/get-support/2-series-bin-expansion.html
  • https://en.wikipedia.org/wiki/Luhn_algorithm

 

Steps to Reproduce (Optional)

Here you indicate the complete and precise steps those lead to the error or defect.

Example:

  1. Open the URL: http://ourwebsite.com
  2. Click on product menu and then choose on product to buy
  3. Click on “Proceed to checkout”
  4. Select pay by Credit Card
  5. Fill out the payment form
  6. Click continue

 

Scope: (optional)

A bug may affect only one part or more than that in a system. You may indicate the scope of error and all of the areas

Example: All payment methods in checkout page except “Wire Transfer”

 

Actual Result and Expected result:

You are reporting a bug because you found something in the system that is deferred in compare of your expectation. For a better and faster feedback from developers or other stakeholders always indicate the current (actual) status and the expected status or values

Example: In the shop report page where there is our monthly revenue report, the final number is wrong. It shows summation of all sales but our developers forgot to subtract refunds. For example, our revenue on March 2017 was 12000$ but we had 1000$ of refund so it should be 11000$

 

Possible cause(s) of error (Optional)

If you can guess for what reasons and causes the bug is raised, you can suggest them here.

Example: the current Credit Card identification algorithm is out of date and needs to be updated

 

Suggestion(s) (Optional)

You can bring any suggestion or solution those could be useful for developers on the table but remember they can ignore them. Sometimes they will even ignore entire bug report by just saying it’s not a bug it’s a feature, a new unexpected feature of course!!

Example: this link could be useful to fix this case: www.braintreepayments.com/blog/supporting-new-mastercard-2-series-bins/

It’s not a bug it’s a feature!

 

Issue Tags or Labels (Optional)

Most issue trackers (Such as Jira, Bugzilla, FogBugz, etc.) have at least one way of grouping issues. Try to use meaningful tags, labels or categories to categorize your cases.

Some sample of good tags is performance, accessibility, usability, UI, admin, horizon, payment, shared server. Do not use tags that won’t indicate cases precisely, such as bug, defect, regression

 

Priority and Severity (Optional)

You can indicate what would be the bug’s priority to be noticed. The options are High, Medium or Low. Also, You can indicate how important and sever is the reported bug.

 

Screenshot (Optional)

A picture is worth a thousand words, but only if it has words to go with it! Most Bug reports should (almost) have a screenshot, but they should (almost) have descriptive text or signs on them.

On the screenshot try to use “Arrow”, “Label”, “Shapes”, etc to clarify your idea and points those a reader should notice to.

There are many commercial and free software available for this aim. I personally prefer “Snagit” software that has every feature you need to capture and modify the screenshots.

It is suggested to use Green color for the correct area(s) and Red color to shows error/should fix area(s). Also, try to save and attach the PNG format since it comes with high quality by default with good compression ratio.

@via: http://ronwhitman.com/it-doesnt-work-and-the-art-of-bug-reports

 

Bug tracking systems:

A bug tracking system or defect tracking system is a software application that keeps track of reported software bugs in software development projects. It may be regarded as a type of issue tracking system.

Many bug tracking systems, such as those used by most open source software projects, allow end-users to enter bug reports directly. Other systems are used only internally in a company or organization doing software development. Typically bug tracking systems are integrated with other software project management applications.

Some of well-known and popular bug tracking systems are JIRA, Fogbugz, Bugzilla, Apache Allura and Fossil.

 

Conclusion:

Good bug reports can save significant amounts of time and money. It is always recommended to define a bug reporting process with some standards and format in your team whether you have a formal QA or not. Don’t risk your time and reputation by just create short and inefficient non-formal bug reports. Investing in a good bug reporting process will increase productivity and developer happiness. Choosing and using one of the mentioned bug tracking systems based on your need can definitely improve quality of your final product.

 

References:

How to properly create a bug report

 

http://blog.capterra.com/top-free-bug-tracking-software

https://en.wikipedia.org/wiki/Bug_tracking_system

https://developer.mozilla.org/en-US/docs/Mozilla/QA/Bug_writing_guidelines

https://shapeshed.com/the-importance-of-a-good-bug-report

JMeter or LoadRunner? That is the Question!

0

I am not a fan of comparison since I believe we cannot just put two things in both pans of a scale and see which one is heavier. Especially when these things got many aspects and features. It almost doesn’t matter we are trying to compare two persons, two technologies, two products or services or even two events. But when it comes to studying things in more details, it would be a good idea to know strength and weakness point of each object from various aspects.

Nowadays in our competitive and rapidly growing IT world, you can find many products and solutions in every small niche market. This diversity got advantages and disadvantages: It’s good because we have many choices and it’s bad exactly for the same reason! Despite shared features and capability each application has its own usage and advantage those make that product unique to users. End-users then can choose among them for the best option based on their needs and their project requirements.

On the other hand, this variety could make users confused and especially when it comes to job market it’s going to be apprehensive. Since you cannot take time and learn all of these tools so you need to choose between these competitors.

In this article, I try to bring two famous and powerful actors in load testing market in a recent decade into boxing Ring: Apache JMeter and HP (Micro Focus) Loadrunner. I will mention their share and different features based on various criteria. At the end, we will have a look at each one’s job market demand based on popular job searching websites in North America: Monster, Indeed, and Workopolis.

JMeter:

JMeter, as I know, is a tool which is used to test and analyze the load on client and server applications, created and developed by Apache Software Foundation. It is an open-source and free software to measure the performance and test the functional behavior. JMeter can run its tests on various platforms both static and dynamic as Java objects, FTP servers, files, servlets, SOAP, databases and queries, Pearl scripts, HTTP, POP3, and much more.

You can use it with many native and 3rd party plugins or even your own developed plugin to gather information and achieve your required functionality. You can integrate it with other tools such as Jenkins, eclipse, and Jira, which allows you to automate the whole of your performance testing.

LoadRunner:

LoadRunner is an automated interactive tool which is used to test the performance of an application. This testing tool is developed by Mercury Interactive to aid in determining the behavior of server and network applications under load normal, stress, and prolonged testing. The LoadRunner performance testing tool was later on taken over by Hewlett-Packard in November 2006. On 7 September 2016, Micro Focus announced its intent to merge with Hewlett Packard Enterprise’s software business segment.

LoadRunner consists of various tools, such as Virtual User Generator or VuGen, Controller, Analysis

Shared points:

Both JMeter and Loadrunner support various protocols and do sampling such as HTTP, FTP, SMTP, JDBC, etc.

Both parties support application environments, databases, and platforms as Web Service, J2EE, .net, ERP/CRM applications from Oracle, SAP, PeopleSoft, and Siebel, streaming and wireless media.

LoadRunner will do most of that and also allow you to GUI test. There are plenty of plugins like SAPGUI, for example, which JMeter does not support. LR is designed to work with/as part of HP ALM/QC, again something JMeter does not offer but many clients use. You can program HPALM to work with JMeter, but it is more work than it is really worth. LR is very expensive and aimed at projects with a larger budget, a complex infrastructure and a longer lifetime.

Their differences:

JMeter Loadrunner
It is developed in 100% Java Loadrunner is developed in C and C++
It is open source tool It is a licensed tool and costly.
Cross platform solution Just works on Microsoft Windows
Can generate unlimited virtual users 50 virtual users in free version
It has huge online community support The online community support is less when compared to JMeter
It does not have JMeter team support if you face any issues. Loadrunner has HP team support when you face issues.
The license of virtual users is unlimited Loadrunner is providing free license, but it is only for 50 users.
JMeter is a single component Loadrunner is a set of components such as Vugen, Controller, and Analysis.
JMeter is very good for web applications. Loadrunner is very good for most of the applications such as the web, java,.net, sap, Siebel etc.
Application business transactions are recorded through JMeter Proxy Application business transactions are recorded through a browser or Loadrunner proxy.
It has less number of protocol support Loadrunner support around 100 protocols including SAP, Siebel etc.
As It is purely developed in Java, it only supports to do customization of the script in Java. The script can be customized in different languages for different protocols such as C, Java etc.
Text validation is supported. In Loadrunner, also text validation supported.
Image validation is supported In Loadrunner, also image validation supported.
Think time feature is available between two transactions Here also think time feature is available.
Pacing feature is available between two iterations. Pacing feature is available between two iterations.
Regenerating of script is not possible It has a feature to regenerate the script by doing changes to the application.
It is supported cloud based testing with Amazon Cloud Services. It is supported cloud based testing by installing the load generators on the cloud.
Monitoring of production servers during the performance testing is not available. You can monitor most of the operating systems, application servers and database servers using Controller
JMeter Reports does not look good when compared with the Loadrunner and the features are also very less. Loadrunner reporting results are very good.
JMeter does not have a feature to integrate with the profiler tools. Loadrunner has the feature to integrate with HP Diagnostics for profiling Java and Siebel applications.
Recording of application is complex. Recording of application is easy when compared with the JMeter.
We get a lot of heap memory issues with the JMeter if we run with a single machine and need to use distributed testing. Loadrunner is stable when compared with the JMeter in terms of running the virtual users in a single machine.
JMeter has compatibility to develop the third party tools as JMeter plugins Loadrunner does not have this feature of SDK.
JMeter does not have network virtualization feature. Loadrunner has the network virtualization feature with the help of third party tool called Shunra network virtualization.
JMeter does not give the breakdown of response time. Loadrunner will give the breakdown of response time.
JMeter supports to test the android applications Loadrunner also supports to test the android application.
It supports to run the load generators in Linux machines Loadrunner also supports to run the load generators in Linux machine.
Parallel execution of users on multiple devices is supported Here also, parallel execution of users is supported.
You will find the Apache documentation here http://jmeter.apache.org/usermanual/ You will find the Loadrunner documentation here http://www.softwarehour.com/loadrunner/

Job Market:

To have an overall estimation on job positions on software performance testing, I have searched keywords “JMeter” and “Loadrunner” in 5 famous job search websites in Canada. Please note:

  1. Some of the results may have indicated both keywords
  2. There could be some replications between these websites (one job position posted in more than one websites)
  3. This job search was Canada wide at September 2017
Job Market Share for Apache JMeter and Loadrunner in Canada – September 2017

Conclusion:

Performance testing is a process of determining speed or effectiveness of a computer, network, software program or device. Both LoadRunner and JMeter can be used for load and performance testing. As mentioned Load Runner is highly developed which has strong support and many features but it has limited ability to monitor server statistics outside Microsoft windows.

JMeter as a suitable tool for web application Performance testing is a very light-weight tool and can be easily installed. Of course, it is free, so no license costs. This tool mainly used for performance testing (load, stress), User can apply automation frame work (data driven, parameter).

Also, job market stats show that JMeter got a higher popularity among users and companies those searching for a load and performance testing. On the other hand, LoadRunner got many clients in banking and financing industry where customer support plays an important role in selecting a software system solution.

I, as a QA engineer, used to work with both tool but I cannot decline that I have a huge crush! on Apache JMeter but still, try to use LoadRunner as a successful and user-friendly software environment.

Other load testing tools:

There are also other tools available to take into consideration which I just name them here:

IBM Rational Performance Tester

Microfocus Silk Performer

SmartBear LoadUI pro

Radview WebLOAD

Sources:

https://www.trustradius.com/compare-products/hp-loadrunner-vs-jmeter

http://www.jmeter.info/2016/01/jmeter-vs-loadrunner-who-is-the-best-in-the-market/

http://jmeter.apache.org

https://software.microfocus.com/en-us/software/loadrunner

https://www.guru99.com/performance-testing-tools.html