COMMERCIAL AND PROFESSIONAL APPLICATION DEVELOPMENT
Part Five - Testing And Debugging

Written by Stéphane Richard (Mystikshadows)

INTRODUCTION:

And here we are, you coded your big application project, you were prepared for it to, and today you are at your desk, patting yourself on the back on a job well done, you sit yourself back and you are about to sip down a fresh cup of coffee. When suddenly you get an email. Not from your boss, from users, people that paid the big bucks for your applicaiton, clients in other words. It seems you either forgot something crucial to the working of FinanceCAD, or they get an error when they try to run it, or install it. And you thought you were done with this project didn't you? You start to stress like never before, you get dizzy and you almost choke on that sip of coffee you were taking when you got the email. Then, you wake up, in your home, and you say to yourself, thank God I was prepared. But the big wheel doesn't stop at the end of the coding phase, far from it. Welcome to the official testing and debugging phase.

Don't worry, bugs as they are called, have been around for a while, no self respecting software had, at one point, a testing and debugging phase. So your project isn't the first, and it certainly won't be the last. In this series we will look at this phase in all it's glory, some say the testing and debugging phase is the most important phase of the development cycle and I tend to agree with that. The main reason is because it directly affects the overall quality of the resulting application and once again, if you've prepared for it (like all other phases of the development cycle) in advance, you can go through the testing and debugging phase in an organized fashion. This is what this part of the series is dedicated to. We'll see what you need to do in order to make this phase as smooth as it can be. So let's get right to it.

WHAT ARE THESE DREADED BUGS:

A bug is a problem, at any level of the application. Typically a bug is noticed first if it stops the normal execution of the application. But there are more than that too. There are different levels of bugs and of course, fixing the bigger bugs is typically the way to go, then you work your way down the list of bug priorities from start to finish. Let's review these categories of bugs, in order of priority, so we can see how bugs can be categorized:

• The Critical Bugs:
  These are of course the highest priority bugs. Critical bugs will, for the most part abruptly stop the execution of the application or create errors in the files they save, or the data they save in the databases. One way or another, these types of bugs will stop the user from using the application as he or she cannot go on using the application until that particular problem is fixed. These bugs include general errors that force the application to end, problems when saving the files or database records, and other critical issues of the sort.


• The Severe Bugs:
  These are pretty important on the priority list because they too, in most cases, will stop the execution of the application and report the error. The difference here is that when the error is reported, the user could start his work over from the start or redo his or her last job. These bugs could include parts of the reports that simply do not calculate because of, for example, a division by zero error.


• The Design Bugs:
  These bugs are usually detected by the people that know the business involved. This type of bug is harder to find for the programmer that got a form to do, did it, but something was missed by the design people for some reason or the problem simply wasn't well explained or well understood. This type of bugs emphasizes the need to take the time and make sure that the design part of the coding phase is present, and everything is understood by both parts before the design issue is approved. If you can do that, then this type of bug can be almost eliminated.


• The Business Logic Bugs:
  This type of bug is usually detected by a tester or user that knows how the business side of the application is supposed to work and what the results of an action should be as per what the expected action or result should be. these are usually bugs in reporting functionality where the result are falsified by a calculation, or a missing result is not present in the report and should be.


• The Important Bugs:
  This is the type of bug that is, as it's name implies, important as in they will need to be fixed, however they don't stop the user from using the application itself. For example, let's say you have a menu option to execute a report, and you also have a toolbar button for the same report and one of these two isn't connected to it's functionality. The user can still create his report using the other alternative so he or she is not blocked by that error.


• The Cosmetic Bugs:
  Well the name says it all really. This usually entails that either a control is not in the right place on the screen, or a term isn't named right as per the design phase or the order of the controls on a form is wrong (when you press the tab key for example). Typically however these bugs do not stop the execution of the program in any way and should be considered lower priority.

I can't stress this enough, but these categories of bugs have to be established before any "efficient" testing and debugging can occur. Users tend to consider any bug as critical, and this is why I am listing them here because there really is a difference between these bugs and their importance. When you receive a bug, the first thing to do is take the time to categorize it properly, just doing that will help greatly with the testing and debugging phase.

Now, when is the best time to start preparing for the testing and debugging phase? As soon as the coding phase begins. Let me explain. There are certain things that can help you waste as little time as possible before you officially enter the testing and debugging. Remember that you have the visual aspect of your application and the non visual aspect and both will need to be tested thouroughly. Let's take the forms to begin with. When you create a form, you should establish, right then and there, how the form should work. By that I mean you should know, from the design part of the coding project, what fields must be entered, what calculations should be performed what happens after the calculation is executed, and the order in which the fields or controls should be accessed in. Once the form is coded, the first test to do is to make sure that the form can perform it's designated task in a normal usage situation (taken from the user requirements and the design part of the coding phase) for all used input devices (when using the keyboard only, or the mouse depending on what you are testing). For added testing, this preliminary testing could be done by another programmer which didn't create the form or the code being tested. Remember that it's good to have a good way for the user to report their bugs, there are many applications out there that allow you to manage the bugs that are discovered as testing begins. For example: There is BugZero™ which is an online web based bug reporting and management tool and BugCollector Pro which also offers a great tool to collect, prioritize and organize the bugs. These tools are made to do just that and the do their job well.

If, in normal situations, you can go through the fields, enter the data, in the right order, and perform the task the form was coded for, you have just created a preliminary test of the form. The preliminary test can help weed out, at the coding phase, almost half of the testing and debugging phase so it's crucial, when planning the project coding phase, to attempt to give enough time to perform this preliminary testing on everything that is coded. Again I can't stress this enough that preparation, at the right time, will greatly affect the testing and debugging phase in a good way. This is true for forms as I've explained, for reports (to test the validity of the results returned by the report, for databases inserts, updates, deletion and querying too for obvious reasons. There are software that allow you to automate a series of tests to be applied to an application or parts of an application. One of the most popular testing software available is called TestDirector™. If you take the time to visit and read about Test Director you'll soon see everything it allows you to do in a testing and debugging phase and you'll quickly see just how good it is at what it does. When a bug is first reported, it's important to know some minimal information about the bug and the situation that caused the bug. In the next section we'll look at exactly what you need to know about a bug.

THE NEEDED INFORMATION ABOUT A BUG:

If a users reports a bug just by saying "the program doesn't work" it doesn't really tell you much about the type of problem that occured and where it happened. It also doesn't give you many clues as to what a possible correction to the bug could be. Just like the other phases of the development cycle, documentation is the key to a successful testing and debugging phase. If enough information is supplied about a reported bug, the time to locate the bug and ultimately fix it finds itself greatly reduced thus shortening the testing and debugging phase. So let's review what type of information that can be provided about a bug that can help you in the testing and debugging phase.

• The Bug Category:
  As mentionned earlier in this document, the first thing to know about a bug is what it's category is when it is reported, and what it ends up being. Like I said, users tend to categorize any bug as a critical bug when really it is not. And since the whole testing and debugging phase is influenced by the type and category of bugs, the only way to gain control over the bug is to give it it's proper category so that it can be managed accordingly. Perhaps a meeting with the users or inhouse testers, to explain the different bug categories, would be wise so that everyone is on the same wave length before starting the testing and debugging process.


• A Title For the Bug:
  We're not talking about a Title or a qualifier for the bug here per se. But a one line, short description of the bug. For example: "Unable to save the design" or "Costing report won't print" and the likes. This helps everyone involved in the testing process get a good clear idea of the problem without needing to go look at the inner details of the bug to figure out what the problem is all about. you, and your users/testers just need to be as descriptive as you/they can in a short sentence.


• Which Form Or Which Action Was Being Performed:
  You'll find that most bugs reported by the users or the testers are at the visual aspect level (except for error in the calculations of a report for example). This means that the user typically selected an option from a menu, which brought a form on the display where the user is expected to do something (this could be a dialog, the main application window and other types of forms too. And an error occured while the user was trying to enter the expected information and either save the information or perform the expected task with the information. When a bug cannot be reproduced, this type of information becomes essential to attempting to track the bug and ultimately reproduce it so it can be corrected.


• Which Control On The Form Caused The Error:
  In a typical scenario, an occurs once the user performed some task, pressed a button, left or entered a specific field, or did anything in his typical normal use of the application. Because coding is typical done at the form level, you can imagine how useful it is to know which form was currently opened by the user. This could mean which form he was working in and which other forms of the application were also opened. Both of these would server different debugging purposes but they are both as important as the other to know. An open form may affect the current form in some way and knowing that it was opened when the problem occured on the other form becomes vital information.


• Which Version Of The Application Was This Bug Found:
  This should be among the first fields to look at. Often in my career I've come accross the situation where a user reports a bug only to find, often after some time is spent trying to reproduce the bug, that it was actually corrected in the current version and the user was using a previous version of the application. For this reason I thought it important to state it here because if it happened to me, it can also happen to you and probably will at some point.


• Can The Bug Be Reproduced:
  This is a very important piece of information to have. Bugs that can be reproduced can be corrected and it is false to assume that any bug can be easily reproduced. Of course, alot of bugs can be reproduced and by reproducing them you are, in some way, confirming the existence of the bug and can go about correcting it. This information means that the user detected the bug, and retried the last actions he or she took to see if the same problem will happen again, if so, then the user should state that the bug can indeed be reproduced. In some cases an error occurs and the user has no idea where the bug came from and how it happened. So it's important to know that the problem did happen, but since it can't be reproduced, we'll need to wait until it happens again to take a good look at it. These are bugs in the bank so to speak, we know they are there, but they have to happen again before they can be looked at.


• The Steps Involved When The Bug Occured:
  This is where the user or tester can really detail everything that happened that he or she thinks is relevant to the problem that occured. If they were in a form, entered all the information pressed the ok button and that's when the error occured, then this gives the programmers a very big lead as to where the error resides and what part of the program potentially caused the error. It doesn't have to be a 50 page explanation of everything that happened in the process of performing a specific task, but the more details that can be provided by the testers and users, the greater the chance of the bug being found, reproduced (even if the user couldn't reproduce it) by the programmer and ultimately corrected. This means that you, your programmers and the testers should meet and make sure that the level of detail in this description be agreed upon by all parties involved. This will help the whole process greatly.


• The Steps Involved In Fixing The Problem:
  This is filled out by the programmer that is fixing the given bug. This helps the programmers learn about the bugs and how they were fixed. In the future, if another bug similar to this one occurs, the programmer can refer to that bug for a possible solution to the problem. Again here the level of detail depends on the bug itself but should be clear enough to clearly understand the steps taken to solve the problem. A bit of common sense is usually all that's needed to provide adequate information.

Of course, if you are managing more than one application, you can add "which application" as the needed information, but that goes without saying. Error reporting is also done from the application in the Error Management part of the code. There you can help yourself even more by adding code specific information to the error report. Code level information would be, for example, which routine was being executed, in which module is the routine found and other such relevant information. This will definitaly help locating the bug at a much faster rate as well as correcting the problem too. Another important point to make is that if there is more than one programmer involved in the testing and debugging phase, these programmers should have some mean of reporting the bug they are working on.

So you have your bug categories, you have all the information you need about a given bug. What should you be doing? This is what the next section will talk about. There is a certain method in the testing and debugging phase, a certain order of things so to speak and because I've seen this order tempered with at different levels in my career, I will explain them here, in the order they should be in for a better and quicker testing and debugging phase.

HOW TO GO ABOUT FIXING A BUG:

As I mentionned, having the information you need to look at a bug is only half the solution. As much as it's important to know of the existence of a bug, it is also important to know what to do with the bug in order to have it corrected hopefully as soon as possible. This is where a systematic method of dealing with a bug becomes important. Of course some types of bug can easily, obviously and quickly be fixed. For example, a cosmetic bug is usually fixed by moving a control to it's proper place, changing the label with the rightful name of the field, or changing the tab order of the controls so that the expected order is achieved when the user presses the tab key to go from one control to the other. Those are quickly located and fixed. However, for the other types of bug, having a method in place can give the programmers a sense of control over the bug and help the whole testing and debugging phase go smoothly. Here is that method, that order of steps to follow when dealing with a bug. Hopefully this will help you deal with the debugging phase in a much more organized and controled environment making it easier to manage.

• Make Sure The User's Environment Is Up To Par:
  If your application takes advantage of specific functionality of a given target operating system and version. These should be stated as a minimal system requirement. So then, if your project is made to work specifically on the Windows™ XP platform and the user is running Windows ME or Windows 98, there isn't much that you can do. Of course if there is a way to perform the same task without requiring a specific version of the operating system, that should be weeded out at the design and coding phase and would typically be prefered over a method that does use an OS specific functionality. This avoids this kind of situation and hence this type of bug. If there is no choice but to use the specific functionality provided by a specific version of the operating system then they should be stated in the system requirements so the users know that they should upgrade their operating system if they are to use your application.


• Reproduce The User's Environment:
  Assuming the user's system does meet the system requirements of the application, it's good practice to have a system that you can continuously format and reinstall with the different operating system that you claim to be compatible with in your system requirements. This way, if a user is using a different but supported operating system you can reproduce his environment and see if the bug can be reproduced in the environment you used on your system and in the user's environment. This helps determine the true nature of a bug and the different alteratives you can pick from to try and correct the problem. Hence it's valuable information to have.


• Reproduce The Incident:
  The bug, as explained by the user, states that it can be reproduced. This doesn't necessarily mean that you can reproduce it on your machine, it means that the user can reproduce it on his machine. So, on your side, the first thing to try is execute the steps mentionned in the details to see if the error will happen for you as it did for him or her. You should always try to reproduce a bug first, it should be the first step to take because if the bug does happen in your system too, you will understand better what the bug is all about. And again if you understand the bug clearly, you can correct it clearly too and probably explain your steps to solving the bug in a clearer fashion for all others to read and understand. The user's description will tell you if he or she used the mouse or the keyboard when the problem occured. If the bug doesn't happen as stated you might want to try it with the other input device just to be on the safe side.


• Locate The Bug In The Source Files:
  This is where code level error reporting can come in handy. But if we assume that only the Error number and description is provided then you'll need to see, from where the error occurs, where in the code it could potentially be located. If you think that the programmer that fixes the bug might not be the programmer that created the faulty code, then it can become an investigative process that needs a minimal time for the new programmer to get familiar with the code he or she will be dealing with to correct the problem, and then see where in that code the error is produced. If you happen to know who created the form or the report that causes the problem and can assign the bug to them, you can save alot of time this way.


• Fix The Bug:
  This goes without saying that once you located where the bug occurs that you should fix it since it's the whole point of the testing and debugging phase. Needless to say that a quick recompile is in order to test to make sure your corrections truely fixes the bug in question. Depending on the company you work for, some may have, in their daily routine a recompile of the whole application at the end of every day and no other compilation is allowed, this isn't necessarily a bad practice as it offers a controled compilation environment, but it does mean you may have to wait until the next day to see if your corrections actually worked. In the mean time, you don't have to just sit there and wait, there are other bugs that you can fix and that means that the next day, when the compilation is done, you'll have more error corrections to check and if they are fixed properly, to finish the debugging process for those bugs.


• Test The Correction thouroughly:
  I've seen this time and time again in my career. A programmer receives a bug from a user, he thinks he knows how to correct it with a certain degree of certainty, fixes what he thinks is the problem and checks everything back into the finished product assuming that the problem is solved. The user rechecks and reports the exact same bug again. When you or your programmers correct a bug, they shouldn't close the bug before they are sure that the correction they brought to the code really does fix the problem. If it doesn't fix the problem on their system it will surely not fix it on the user's system either. Simple common sense again will minimize the number of reoccuring bugs in your application and will help the whole testing and debugging phase move forward. For this reason I thought it important to mention it here.

And there you have it. As I mentionned, it seems that because of time issues, these steps aren't always respected and often it's the cause of a never ending testing and debugging phase that just seems to get longer and longer. So before you decide you should follow these steps or you think you don't need to, take the time to think about it a while. Good testing and debugging practice will help the phase reach its end and in the world of software development, the end of the debugging phase should be of the highest priority throughout the development cycle.

Even if the size of the project is much much smaller than a big scale application, testing and debugging should not be neglected or overlooked at all. In fact, in many cases, not paying attention to this phase is often the cause of delayed releases, deadline that can't be met, and other serious issues of the sort. So for these reasons, testing and debugging must be treated as a very important aspect of the development cycle.

AND NOW FOR A FINAL WORD:

And this pretty much concludes the whole testing and debugging phase. Needless to say there's alot of material in this phase too. Some of you might have discovered many things about the testing and debugging phase that you never knew existed. If this is the case, then I'm glad because it means that this series of documents is doing it's job. This is the reason why I created this whole series of documents, to really teach you what you need to know about each phase of the development cycle and what exists to help you along the way at each and every step of the way. I also hope that I've made clear, in each part of the series so far, that preparation is as always the keep to the success of each of those phases. Some of you may wonder if there is really a need to do absolutely everything that I am mentionning in this series. Well to that I can tell you that it won't hurt to do them all, it will make the whole project documented a lot better. Maybe, after a few projects you'll think you're familiar enough with the process to start creating shortcuts to these steps. Well, what I can tell you about this is that I wouldn't if I were you. Just like any process you do in a job. If you have all the documentation you should have, then there is no guess work involved and in a development project, the less guess work you have, the better it is for the life, and the success, of the development project.

There is one more part to this series, Installation and Integration. That part will talk about the last steps of the development phase, making sure your application installs itself and executes as it is supposed to. there's different considerations for a commercial and a professional application and we'll be covering each of them in detail in the next and final part of this series. Until then, email me with questions, concerns, suggestions and ideas about this series of documents if you want to know more about a certain part I've covered in this series, let me know as well and I'll see how I can better the contents of this series. Until next time, happy reading and I'll see you then.

MystikShadows
Stéphane Richard
srichard@adaworld.com