JIRA Test Management

Generally speaking, JIRA (from Atlassian) sucks for test case management. I hope I’m not telling you something revolutionary there. Unfortunately what I’ve seen this lead to is for testers (and leaders and managers) to decide the best solution is to simply use QC or ALM. A solution looking for a problem bound to lead to a need for more solutions if I’ve ever seen one. To those wanting to do this: No. Stop it. Bad manager. No cookie.

It might be less of a problem given that HP is now making connectors for a number of IDEs to allow QC/ALM to talk to everything in Development land, but cost is usually the number one complaint with HP’s testing products.

There are easier (not to mention VAHHHHHSTLY cheaper) options available. Instead of going down the HP path to financial ruin, consider using Zephyr, available from the Atlassian Marketplace. There’s a nice eight and half minute demo video available at YouTube that will give you a quick intro to its capabilities.

Combine this with Bonfire for session based testing and you have a pretty powerful suite of tools for quite a reasonable price (Zephyr starts at $10 for 10 users – this includes maintenance and support for a year).

Tagged , , , , , ,

What’s this Agile thing and how do I test in it?

There’s quite a bit of work at my current employer regarding agile testing. What some might see as a part of the overall test management process others see as a discipline requiring its own focus. I’m getting to the point where I don’t give a shit (I originally got a little ragey about the whole thing which wasn’t helping so I decided that if you can’t beat ‘em… arrange to have them beaten) but I think what matters is how testers fit into an agile team. I don’t think we need to be focusing on what Scrum is or how a retrospective works or what do we do with these cards with all the writing on them. Instead we should be focusing on getting shit done. How do we do that? Try, fail, try something different, succeed, refine, repeat.

What should be of most interest to testers is how do we go about testing in a team that’s doing some variant that at least looks a little like Agile rather than waterfall. Given this context, I’m always on the lookout for anything to make the job a little easier. Rather than reinventing the wheel I keep an eye out for what smart people have already done and look for ways to adapt it.

The good folks over at Ministry of Testing have put together a set of useful resources. One of these is the image below:

agile testing mindmap

There’s a lot to absorb there. The image is worth printing out on a large sheet of paper and sticking to a wall somewhere close at hand when you start looking at the project from a testing point of view. It needs to be easy to get to because you’ll be adding your own notes to it sooner rather than later.

I’m tempted to rush into this and talk about areas that interest me specifically, but I think that’s for another post.

The mindmap even goes into

Tagged , ,

To be or not to be a technical tester?

This isn’t really a question for which there are two responses. In the current era of easy automation toolhahahahaha, okay, you got me, I lied, but anyway… In the current era of tools that provide easy access to basic automation, testers can’t really afford to not be anything other than technical testers. Unfortunately, I keep running in to testers (both at work and online) who have done nothing more than test web based applications for the last x years and don’t understand the first thing about what lies under the hood. Of anything.

I’ll put it in simple terms: If you aren’t technical, your shelf life is limited to about 5 years. Your career will not advance if you don’t educate and challenge yourself.

My current rant stems from a couple of posts on LinkedIn, one recent, the other not so much. The more recent one asked:

What are the “good to have” technical skills for a tester?

I replied with a quick list of skills I’ve developed over the years and found useful:

  • Basic coding knowledge and principles
  • Network architecture and transmission stack
  • Good understanding of
    • physical hardware
    • database structure
    • basic and intermediate SQL queries
  • Multiple protocols
  • Firewalls
  • Good working knowledge of the linux command line
  • Some degree of knowledge about security testing and basic attacks
  • Knowkedge of a good variety of testing tools and more importantly why you’d use them and where (not just QTP or JMeter but tools like traffic analysers)

These are on top of general testing skills in specific domains (both functional and non-functional) and the tools that go along with them. Obviously I’m not counting the non-technical skills that also play a oart in being a well rounded tester. Another response covered this with a partial list:

  • Project management skills
  • Systems theory, testing theory, test methodology
  • Communication skills
  • Critical thinking and reasoning skills
  • Scientific, creative and investigative type skills

There are no doubt more to add to both lists. If you can come up with more, please feel free to add them in the comments.

The latter source of my rant was a post some time ago in one of the performance testing groups. Someone without much of a clue on what’s inolved wrote:

We are about to start teaching “Performance testing” to a bunch of freshers. By default we cover LoadRunner(Scripting and Execution). What else can be covered to ensure that they don’t fall in the trap of ‘Performance Scripting’ only? Remember that the target audience are fresh engineering college grads (IT and non IT)

There are so many things wrong with this I don’t know where to start. Fortunately there are brighter minds than I, one such being James Pulley (from PerfBytes; he’s also the CTO of NewCOE) who responded with the following:

You should not expect to spend any less time becoming a performance tester than you would a plumber, an electrician, a barber or a cosmetologist. In my state it takes around 1800 hours of combined training and internship to become a cosmetologist, so let’s use that figure as a starting point.

Since you will have a set of team members whose fundamentals are in question and all over the map in terms of maturity, spend the first 600 hours concentrating on those core foundation skills which are completely independent of performance testing: Programming in the language(s) of your tools, application architecture, operating system architecture and implementation, database fundamentials, networking, systems analysis, requirements management, project management, testing concepts (independent of software testing and/or performance testing), the scientific method, etc,….

Now that you have brought everyone to a common foundation level you can begin introducing process and theory around performance testing. Begin with the process items of planning and analysis, reinforcing the foundation skills covered earlier. Continue with the tools. Begin with JMETER, a pretty basic tool, to ground the concepts and then move to more sophisticated commercial tools. If you have not paid for a training license for the commercial tools then do not use them at all. Assume that this will consume the next 600 hours of time.

Then it comes to a time to bring together all of the foundation skills, the new skills, the tool skills, etc,,, into a cohesive whole. if you simply hand the person a briefcase and parachute and send them off to a gig then they will fail. They should spend the next 600 hours (at least, if not significantly longer) under the internship of a master in the profession. You should have defined goal posts in terms of professional development before people can move on within the internship or exit from it. Failure to hit a goalpost either sends you back to training, or into a different profession after a period of time and retry.

I thought it was a great response and helps to highlight one of the root problems I see with so many questions on the LinkedIn testing forums. There’s the assumption inbuilt that skill with a tool is the panacea to ones career problems, the ticket out of doing manual testing. It’s not. Without a thorough understanding of the theory behind these tools all you’ll ever be is just another button presser. Is that what you really want?

Tagged , , , ,

How do I test this?

Another fun one from the bottom of the barrel that is the LinkedIn Software Testing and Quality Assurance discussion group.

Mr K writes:

What do I test when I’m testing the product?

Well Mr. N, in the short time remaining to you in your current role, before your manager realises what a complete waste of space he’s hired in the form of “Senior Testing QA Engineer” I don’t have enough time to answer you in any form that will really satisfy you. Mainly because the answer will still not write the test cases for you, which is what you’re really looking for. Don’t lie. Anyone reading your question would know you’re an incompetent tit.

However, for those of you who actually are interested in just how much testing you probably didn’t do, someone tweeted about Michael Hunter’s blog thebraidytester. Pretty good read and LOADS of good stuff about testing. But for truly scary reading, check out the You’re Not Done Yet list of… Well, list of things you missed pretty much. Look on the bright side though: others have missed the same stuff before you.

Here’s another (slightly shorter) one from Elisabeth Hendrickson (@testobsessed). It’s a quick reference test heuristics cheat sheet. Print it out double sided and laminate it. Keep it handy when reading requirements or looking at user stories too. It’ll push you to ask interesting questions, which is pretty much our job description.

I’d like to recommend the source of those links too. Aleksis Tulonen (@al3ksis) put together a quick slideshare on exploratory testing and included the above cheat sheets as well as another that I haven’t had a chance to look at yet. Go read his blog too – some great stuff there.

Tagged , ,

LinkedIn, how I loathe thee

Okay, I’m just about done with the LinkedIn discussion group Software Testing and Quality Assurance. I feel like losing my shit over some of these dumb as fuck questions. The low point came today after some cretin asked “What’s the difference between pre and post production testing?”

Other days it’s been great demonstrations of unemployability like “What the difference between system and SIT testing?”, the old standby “What’s the difference between testing and QA” and one of my personal hates “How do I estimate testing?”

Even better is the “Should I get ISTQB certified?” or one of today’s leading lights “Offshore testing or in house – which is better?” which is just like asking “What do you like to eat?” and about as fucking useful.

I used to respond to these mind numbingly simple questions. “Learn to use Google” or “Asked before. Use the search tool” but I’ve pretty much given up on that. As to the ISTQB question “I can think of better things to wipe my arse with” wouldn’t be acceptable apparently so I try to find nicer words that make the same point.

Honestly, how the people asking these questions can call themselves software testing professionals is simply beyond me. I can only assume that their employer isn’t looking at the posts on LinkedIn. If they are and aren’t drawing up plans for finding something to fill their existing role (a houseplant comes to mind) they deserve everything they get.

If it weren’t for some of the more interesting people (if you’re reading this, you know who you are. And aren’t) I’d stay as far away from the discussion as possible.

Tagged ,

Testing a postcode lookup function

It sounds simple enough, but what are the test cases when testing a postcode lookup function? This article is to give you some ideas on the type of tests you should perform to find the major defects.

When testing this function, consider your target audience. For example, if your software product is in the retail sector and the customers will be searching a delivery address, you will need to understand where your customer base is. If you are based in the USA and your target audience is always going to be in the USA, then testing UK postcodes is probably something which should only be done as a negative test.

You will also require some basic negative tests to stretch the functionality and ensure validation is done and that any error messages which are returned make sense to the user.

 

Your target addresses

- Mainland addresses. Insert ‘normal’ addresses which you would definitely expect to be validated successfully.

- LocalIslands. For the UK, you may wish to test postcodes in the Isle of Man, for example. For Australia, you may wish to test postcodes from differing states and territories.

- Local countries within scope. If your audience will be based in local, in scope, countries, create some tests for them. For example, if in the UK, you may wish to test for Ireland and Northern Ireland.

- Compare results with a Post Office locator. The UK Post Office has a locator on their website (found at http://www.postoffice.co.uk/postcode-finder). This can be used to double check that the correct postcodes and addresses are returned by your function.

 

 

Negative cases

- Invalid addresses. Using a correct format, enter a postcode which does not exist. For example, in the UK you may wish to try “AA1 1AA”. This will return no results.

- Strange postcodes. The UK has special postcodes for certain departments. For example, Army Bases get their own postcode (an example might be (“BFPO 11”)

- Invalid Characters. This is a fairly obvious one but try to insert spaces, special characters and an invalid format.

 

Tagged , , , , , ,

JMeter Framework

I’ve started a bit of a personal project to build a framework for load testing using the open source tool JMeter. The framework will be based around Pentaho’s community edition of Business Analytics called Kettle. The aim is to have Kettle handle the loading of logs into a MySQL DB. Might be overkill to use Kettle when I could just laod from the command line but Kettle gives me more options down the track for loading data from other sources. Once I have the JMeter > MySQL problem licked I’ll move onto getting perfmon logs added as well for performance testing on Windows. Linux shouldn’t be as much of an issue as I’m a little more familiar with system logging on that platform.

I’m in the process of collecting useful links, but initial tool selction can be found here:

I’ve also downloaded for later use:

Not sure what to do with these yet but I’ll take a look at them after the data loading.

Some other useful resources I’ve been looking at:

More to follow as I discover stuff.

Tagged , ,

Program Managers take note

What to expect of your test managers. Great article by Peter Farrell-Vinay. It arose out of a LinkedIn discussion about what senior management would or should be seeing from the testing department in terms of reporting.

Peter added:

  • how many requirements
  • how many requirements not covered by at least one positive and negative test
  • how many complete set of tests are executed.
  • how many tests cannot be run because they are blocked by a bug
  • what percentage of features are completely bug-free
  • how many bugs are outstanding for > 1 month
  • how many bugs are allegedly fixed, in a current release, but not yet tested?

Good starting point for anyone.

Providing Feedback

When you’re asked to provide feedback for someone, it can be a difficult task. Whilst you don’t want to completely destroy someone with your words, equally you wouldn’t want to inflate their ego so much that they decide that they are worth more than they are appreciated for – and leave for another company!

There are a few different methods to giving useful feedback, my favourite being the ‘muck sandwich’ (a word has been altered to avoid swearing!). I will explain this method in a moment.

In my opinion, you need to provide various pieces of information in your feedback. Regardless of your method, I believe that the essential items to be included in feedback are as follows:-

  • Detail behaviours which are seen as positive. Things like motivation, attitude and professionalism.
  • Detail skills which the staff member shows particular strength in. This could be things like analysis, attention to detail and organisation.
  • Detail areas of improvement. This is commonly the most difficult area to feed back on.
  • Overall comments.

Providing examples of behaviours or skills, whether good or bad, can also be useful as this can help support your feedback. Be careful with this though as it may cause some offence, depending on the audience.

The ‘muck sandwich’ method is a nice way of delivering bad news as the feedback layout almost conceals the area of improvement neatly between two happy smiley paragraphs. The ‘muck sandwich’ method consists of three paragraphs.

Firstly, deliver a good story / positive news.

Secondly, deliver the bad or not so good news / area of improvement.

Thirdly, finish on a lighter, more positive, note.

Here’s an example (not based on anyone real – although if they do, it is purely coincidence!):-

“Dave is an excellent tester as he managed to find a large number of critical problems in our Business Processes during the testing.

He also shows an ability to raise defects quickly but this can sometimes cause him to make errors when writing the defect reports. A slower, more methodical approach might help him improve this side of his work.

Dave’s organisation skills and polite communication approach really help him get what he needs from our clients.”

The above solution is for more generic feedback but it is always best to align feedback to core KPI’s if they exist. This way it is easier to detail constructive and positive feedback, which targets the exact areas that are being measured for performance.

The vital point on feedback is to actually do it! Don’t wait for someone to ask before providing it – and don’t be afraid to ask if it is not forthcoming!

Happy feedbacking!

Tagged , , , , ,

Jamie Lees – 6 Random Questions

I came up with a list of my own 6 random questions for Jamie. Enjoy :)

Hi Jamie. Welcome to 6 of the best by Breaking-Software. This is where I ask you 6 questions and you give us your answers in your own unique way.

Breaking: What are the top three things that give you the shits about testing?

JL: Three main things:

- The main thing is some people’s lack of respect for the test team. Inexperienced Dev’s and Managers see Testing as a complication and a pain in the arse, but it is the most critical and important phase that a project goes through.

- Incorrect attitude. I mean this in terms of your test team, but really it applies to anyone. If someone is new and has the right attitude, they will go far and learn the things they need to learn. Someone can have all the experience and skill in the world but without the right attitude, they’ll go nowhere.

- The question: “Why didn’t you find this defect?”. Well, why do you think?! More than anything, this demonstrates a lack of understanding of what the Test team do. Pointing the finger ‘gives me the shits’.

Breaking: What don’t you like about Agile?

JL: This isn’t a problem with agile as such, it’s more a problem with the people who think they are doing things in an agile way…but clearly aren’t. Just because we talk about burndown and have a SCRUM, this doesn’t mean we are agile. This in itself is a flaw of agile, as it appears to be open to differing interpretations. Whereas Waterfall is Waterfall. I think we all know what that entails. Otherwise, I see massive benefits to operating in an agile way. The method puts the customer first, which is absolutely the right way to approach delivery of software in most domains.

Breaking: Do you think testers need to be more technical or more business focused to survive long term in the industry? By technical I mean knowing the application stacks from different vendors, knowing how to configure a test environment, deploy a specific operating system for testing on a virtual machine, possess some coding skills, set up a build server, etc.

JL: Oh, I’ll slice right down the middle here, as you might have predicted! To deliver a reasonably complex program of work, the test team need to be able to wear different hats. If they can’t, you’ll physically need the two very different skill sets in your test team.

Breaking: Name the top four skills/traits/abilities that a good project manager possesses.

JL: I’d say:

- Task oriented. This helps focus on one thing at a time and on details. Focus on this will mean you uncover more risks / issues prior to delivery.

- Experienced. Of course experience doesn’t necessarily mean ‘good’, but it can help.

- Attitude (see question 1 answer!)

- Flexible. Dates and tasks will change and some people won’t deliver what they said they will when they say they will. A good PM needs to be adaptable to this and plan for it.

Breaking: Repeat the same for a test manager and why they differ.

JL: As my answers are reasonably generic (and can be applied to many roles) I would argue that they are very similar. All of the items in the list above also apply for a Test Manager. I would perhaps change the ‘Experienced’ answer slightly but I think it is very useful if a Test Manager understands test techniques and how and when they may be used, in order to help his own team grow and learn from them.

Breaking: What curly/tricky questions do you ask of potential new hires during interviews for a testing or lead role? Give me your top five.

JL: For this question I think its useful to take it back to basics in order to understand if the candidate understands the basics…!! I’ve given 7 though. So sue me.

- Walk me through how to write a test case.

- How do you teach someone testing? (Someone who doesn’t even know testing exists)

- You have 2 testers in your team (one more technical and one more business / GUI focused). Your project testing should be split into two streams, but both require a more technical type tester. What do you do?

- You’ve done your test estimates / planning poker. Your Product Owner thinks they are wildly inaccurate. What do you do?

- You have requirements in your project which focus on Security of the system. As the TL, what do you do with these requirements / stories?

- When managing a test team of people who aren’t based in your office (offshore / outsourced), what do you find is the best approach the getting the best possible outcomes?

- What does ‘focus on risk’ mean to you?

====

That’s it for now. I’m going to hit up some other contacts for more random questions.