I'm currently reading "Lessons Learned in Software Testing" by Cem Kaner, James Bach and Brett Petticord. I read their blogs and I'm pretty sure I've seen at least one of them give presentations at STAREast before. They're pretty much considered the gurus in the QA field and I respect them a great deal.
Their recent book is a quick read and its format is such that you can pretty much pick any point in the book and start from there. I like it because you can skip the pieces that aren't applicable to you and simply go to the sections that you feel are applicable. They're organized into 'lessons' and are pretty much independent of one another. There are 293 lessons in total. I always enjoy reading their work because it's authored in a way that's applicable to most software companies that have people filling the roles of Product Managers, Project Managers, Development Managers, QA Managers, etc... While this is all well and good at established companies, what do you do at a startup that maybe only has 6-20 people total in the company?
Many of the lessons in this book refer to the role of Project Manager. What if your startup doesn't have a PM or someone who's sole responsibility it is to make sure that releases get done on time and with all the whiz-bang features in them? Where do you, insane-startup-QA guy (or gal) go for your guidance? How do you adapt your processes to Project Manager's when there isn't one?
Well, I don't claim to have all the answers, but I'll share some of my own lessons learned as they apply to startup QA. I'll probably just cover one lesson per post in order to keep it simple and focused and invite some comments.
Lesson - If they're using XPAgileFall, then use XPAgileFall.
I interviewed one time at a startup and was talking with one of the developers. I asked him how he would describe their software development life cycle. His reply was simply "We don't have one." I think he was waiting to see if a look of astonishment appeared on my face. Instead I just laughed and replied that it wasn't the first startup I've seen that follows the W.D.H.O. standard.
One of the first things I try to do is to draw out whatever process I observe them following. It probably won't fit into any popular model, so what, it doesn't have to. Even if they claim that there is no process, there is, they just don't have a name for it or have it documented or follow it repeatedly. It just has to be clear to you on how a feature goes from concept to code so you can advocate different types of testing along the way and identify how to interject yourself into this process so you can be 'in the loop' when the critical decisions are made. I map it out because I prefer to see things visually v. reading it textually.
Here's an example of what it could look like (you may have to click the image to see it in all its glory).
Ok. We have a process folks, whether they like to acknowledge it or not, it's there. Now, how can you squeeze yourself into this cycle in a manner that you can be most productive?Let's walk through each phase and figure out how we can help.
Concept/New Feature
Unless you were hired as a Product Manager/Quality Assurance Engineer, it's best to leave this to someone more qualified. It's not your job to determine the product road map or decide what features are needed to meet/support the business plan.
Round Table Discussions
This is where the idea gets pitched to the developers and typically where you can get your first glimpse into the actual requirements for a new feature. If you haven't set up a wiki yet, now's the time. Transcribe whatever notes you have from this meeting there and then send around, ask people to add & clarify anything that's missing. People will use a wiki once they see how easy it is to add content and collaborate on it.
Prototyping
You're not going to build it, but you can offer to test it. Just remember, when you're testing, it's not going straight to production, it just needs to work enough for the purposes of a demo so feedback can be obtained. This will give you another glimpse into how the feature(s) will work. Add your observations to the wiki and invite review & comments from the developers. This is a good point in time to formulate your test strategy and share it with the developers. Let them know how you're going to test and seek feedback from them on how they can build unit tests or harnesses that can help you. I don't typically log bugs at this stage, instead I just send a list of issues to the developers so they're aware of them.
Demo
You can glean a lot from sitting on demos. You can pick up how well sponsors articulate their requirements. You can get more information on what they expect a feature to do. You'll also find out what needs to be corrected on the dev-side of the house so you're not caught by surprise when you see a significant change in a future build of the code.
Integration
This is where the sponsor says "it's great, when can I have it?". Now dev realizes they have to get it to work with the rest of the app so they drop it into the build and hope for the best. Well, you can certainly offer a lot in this stage and you certainly don't want the untested code going out into your production environment. Hopefully, by now, you've built some kind of test server and you can walk the feature through its paces there and smoke test that nothing else was broken as a result of adding the new feature. Developers, ideally, should have unit tests that run with the build and you should be able to execute your test plan (note - your test plan does not have to 100 pages of granular-level tests, keep them simple because it's very likely that things will change and you can't afford to spend time refactoring all of your tests). At a minimum, you want to at least validate the features work based on the requirements that have been assembled and you want to make sure the overall system is functional.
NEW STEP - U.A.T. (User Acceptance Testing)
Once you've got a solid build and you've run through your testing on the feature and you feel that the application is pretty stable, point the sponsor and/or other stakeholders at your test server(s) and ask them to exercise the build too. You'll find that they'll want to make changes to the features because it's not what they expected it to be in the integrated environment or it's missing this or that...that's ok...don't panic. Just get the changes up on the wiki and log cases in the bug db (if that's all you have) or your project management system and rinse & repeat until you nail it and it's 'accepted'.
Production
If you think you can catch every bug you're wrong. I agree with Cem, James and Bret on one particular point from their book. Never be the gatekeeper! This basic premise is that unless you want to have total responsibility for every piece of code that goes to production, don't demand that you stop a release "in the name of quality!". Present your findings to those who are empowered to make those decisions. Advocate quality, but don't use it as a threat because you're mad some bugs aren't getting fixed.
Wrap it up already!
Ok, ok. So you've been through your first release cycle. It probably wasn't all that bad and it you probably feel that there were 20 million other things that you could've, would've, should've done if you had the time and an army of QA clones to help you. Well, here comes that word "reality" again...you don't, but now you've got more information for the next go-round and now people are beginning to collaborate and share the knowledge. It's hopefully on your wiki now, people know where to find it, they know they can actively contribute and they now know to include you in future meetings that are important to QA. Your XPAgileFall process is now evolving and that is all you can ask for at this stage. With time, the team will see the advantages of a collaborative approach to software development and you'll be able to further evolve as you grow.
You'll start to be able to identify areas you need more resources and/or help from development and you'll be able to at least start to answer the question of what was tested more accurately.
I'm sure I left out a ton of stuff...it's a broad topic...I'll dig into more later on.
Happy New Year and see ya in '08.
