Insane Startup QA

XPAgileFall and Process Discovery

2007-12-21T09:44:00.001-05:00

I'm currently reading "Lessons Learned in Software Testing" by Cem Kaner, James Bach and Brett Petticord. I read their blogs and I'm pretty sure I've seen at least one of them give presentations at STAREast before. They're pretty much considered the gurus in the QA field and I respect them a great deal.
Their recent book is a quick read and its format is such that you can pretty much pick any point in the book and start from there. I like it because you can skip the pieces that aren't applicable to you and simply go to the sections that you feel are applicable. They're organized into 'lessons' and are pretty much independent of one another. There are 293 lessons in total. I always enjoy reading their work because it's authored in a way that's applicable to most software companies that have people filling the roles of Product Managers, Project Managers, Development Managers, QA Managers, etc... While this is all well and good at established companies, what do you do at a startup that maybe only has 6-20 people total in the company?

Many of the lessons in this book refer to the role of Project Manager. What if your startup doesn't have a PM or someone who's sole responsibility it is to make sure that releases get done on time and with all the whiz-bang features in them? Where do you, insane-startup-QA guy (or gal) go for your guidance? How do you adapt your processes to Project Manager's when there isn't one?

Well, I don't claim to have all the answers, but I'll share some of my own lessons learned as they apply to startup QA. I'll probably just cover one lesson per post in order to keep it simple and focused and invite some comments.

Lesson - If they're using XPAgileFall, then use XPAgileFall.

I interviewed one time at a startup and was talking with one of the developers. I asked him how he would describe their software development life cycle. His reply was simply "We don't have one." I think he was waiting to see if a look of astonishment appeared on my face. Instead I just laughed and replied that it wasn't the first startup I've seen that follows the W.D.H.O. standard.
One of the first things I try to do is to draw out whatever process I observe them following. It probably won't fit into any popular model, so what, it doesn't have to. Even if they claim that there is no process, there is, they just don't have a name for it or have it documented or follow it repeatedly. It just has to be clear to you on how a feature goes from concept to code so you can advocate different types of testing along the way and identify how to interject yourself into this process so you can be 'in the loop' when the critical decisions are made. I map it out because I prefer to see things visually v. reading it textually.

Here's an example of what it could look like (you may have to click the image to see it in all its glory).
Ok. We have a process folks, whether they like to acknowledge it or not, it's there. Now, how can you squeeze yourself into this cycle in a manner that you can be most productive?

Let's walk through each phase and figure out how we can help.

Concept/New Feature
Unless you were hired as a Product Manager/Quality Assurance Engineer, it's best to leave this to someone more qualified. It's not your job to determine the product road map or decide what features are needed to meet/support the business plan.

Round Table Discussions
This is where the idea gets pitched to the developers and typically where you can get your first glimpse into the actual requirements for a new feature. If you haven't set up a wiki yet, now's the time. Transcribe whatever notes you have from this meeting there and then send around, ask people to add & clarify anything that's missing. People will use a wiki once they see how easy it is to add content and collaborate on it.

Prototyping
You're not going to build it, but you can offer to test it. Just remember, when you're testing, it's not going straight to production, it just needs to work enough for the purposes of a demo so feedback can be obtained. This will give you another glimpse into how the feature(s) will work. Add your observations to the wiki and invite review & comments from the developers. This is a good point in time to formulate your test strategy and share it with the developers. Let them know how you're going to test and seek feedback from them on how they can build unit tests or harnesses that can help you. I don't typically log bugs at this stage, instead I just send a list of issues to the developers so they're aware of them.

Demo
You can glean a lot from sitting on demos. You can pick up how well sponsors articulate their requirements. You can get more information on what they expect a feature to do. You'll also find out what needs to be corrected on the dev-side of the house so you're not caught by surprise when you see a significant change in a future build of the code.

Integration
This is where the sponsor says "it's great, when can I have it?". Now dev realizes they have to get it to work with the rest of the app so they drop it into the build and hope for the best. Well, you can certainly offer a lot in this stage and you certainly don't want the untested code going out into your production environment. Hopefully, by now, you've built some kind of test server and you can walk the feature through its paces there and smoke test that nothing else was broken as a result of adding the new feature. Developers, ideally, should have unit tests that run with the build and you should be able to execute your test plan (note - your test plan does not have to 100 pages of granular-level tests, keep them simple because it's very likely that things will change and you can't afford to spend time refactoring all of your tests). At a minimum, you want to at least validate the features work based on the requirements that have been assembled and you want to make sure the overall system is functional.

NEW STEP - U.A.T. (User Acceptance Testing)
Once you've got a solid build and you've run through your testing on the feature and you feel that the application is pretty stable, point the sponsor and/or other stakeholders at your test server(s) and ask them to exercise the build too. You'll find that they'll want to make changes to the features because it's not what they expected it to be in the integrated environment or it's missing this or that...that's ok...don't panic. Just get the changes up on the wiki and log cases in the bug db (if that's all you have) or your project management system and rinse & repeat until you nail it and it's 'accepted'.

Production
If you think you can catch every bug you're wrong. I agree with Cem, James and Bret on one particular point from their book. Never be the gatekeeper! This basic premise is that unless you want to have total responsibility for every piece of code that goes to production, don't demand that you stop a release "in the name of quality!". Present your findings to those who are empowered to make those decisions. Advocate quality, but don't use it as a threat because you're mad some bugs aren't getting fixed.

Wrap it up already!
Ok, ok. So you've been through your first release cycle. It probably wasn't all that bad and it you probably feel that there were 20 million other things that you could've, would've, should've done if you had the time and an army of QA clones to help you. Well, here comes that word "reality" again...you don't, but now you've got more information for the next go-round and now people are beginning to collaborate and share the knowledge. It's hopefully on your wiki now, people know where to find it, they know they can actively contribute and they now know to include you in future meetings that are important to QA. Your XPAgileFall process is now evolving and that is all you can ask for at this stage. With time, the team will see the advantages of a collaborative approach to software development and you'll be able to further evolve as you grow.

You'll start to be able to identify areas you need more resources and/or help from development and you'll be able to at least start to answer the question of what was tested more accurately.

I'm sure I left out a ton of stuff...it's a broad topic...I'll dig into more later on.

Happy New Year and see ya in '08.

Intro to Insane Startup QA (Part II)

2007-12-17T19:57:00.000-05:00

Thanks for checking back in.

If you recall we last left our hero on the doorstep of an Internet startup company that had brought him in to "QA" their product. Well, what they really meant was they wanted someone to come in and test like heck for two months and find as many issues as humanly possible and get them fixed. Silly rabbits. QA is far more than just testing and verifying bugs as those "in the know" know. Is know know a no-no? Hmm, weird.

Anyways, QA encompasses requirements reviews, risk analysis, project coordination, test plans, automation, metrics, blahblahblah...there are many other websites that can tell you what it is and isn't, I won't bore you with a lecture or argue with you about what the differences are between QA'ers and testers. For the purpose of our story, our QA guy found himself in a situation that called for rapid assessment and exploratory testing. Real QA will come later, once the bodies have been hidden.

Rapid exploratory assessment testing...huh?

Not quite. Simply put, you have to understand what type of environment it is that you're in and figure out what approach you're going to take in order to be effective. You see, time is not on your side. Here are a some tips that can help you with the assessment part...but first, a quick joke...What does the acronym SPECS stand for? Simple Papers Engineers Can't Supply. (Haha...ok, I'm done, no more dumb jokes)...here we go...

Where's the info?

What development methodology are they attempting to follow? XP, Agile, Waterfall (gasp!) or the chaotic hybrid, XPAgileFall? This will clue you in quickly on how organized the team is (or isn't) and help you identify areas where subtle improvements can have a massive impact on productivity.

Ask the question...how does a feature originate, get through dev and then make it's way into the application?

Do they have any documented processes for checking in code, doing builds, running unit tests, deploying code, etc...? More importantly, do they follow them if they do?

Do they use a Wiki? If so, dig into it. Read and comment on pages that aren't clear to you (a guy or gal that knows nothing about the app yet). If not, get one set up a.s.a.p. It's a valuable tool.

Go through whatever they're using for a bugdb (this will be very educational as it will illustrate how others on the team communicate). 10 to 1 says it's doubling as their requirements tracking system too (and that's only ok if it's JIRA).

What kind of environment are you going to have to test in? Believe it or not, you may be testing in a production environment at first, don't be shocked.

State of the application

This is where exploratory testing comes in. You will absolutely have to seek out someone to walk you through the basics of the application. After that, it's time to skin your knees on the app a bit. I believe this definition works just fine for exploratory testing.
You will have to come to your own conclusions here. Typically a lot of the bugs that are caught at this stage in a startup are found by end-users. Maybe the CEO, VP of Marketing, CTO or whoever else outside of development has been hitting the app prior to you getting there. Chances are great that whatever bugs are in the system might be one-liners and/or just a screen shot. Don't get discouraged, my advice is to move on past them until you're more comfortable with the app. Just remember, not everyone can write up a kick-ass bug report like you can.

Who is in play?

This is one that you typically only learn by experience. Every company is unique. But in startups, there is typically an undocumented (like everything else) hierarchy. Here's how it may work...

Founder(s) - They call the shots and they damn well should get to, after all, it's their company. They're the visionaries, you should spend any time you can with them to get their input on what's most important to them.
The First Born - These are the ones the founder(s) first hired to help them. These folks have a tremendous amount of pull when it comes to adding features and are also very in tune with the business, the market and the product. Seek out their help with testing. They're the ones that have been or will have to demo the app. They have a vested interest in helping this app & company succeed. These are often the most valuable testers as they use the application (hopefully) as much as anyone.
The Center of the Universe - Every startup has one. It's that one person that everyone goes to for answers or is there keeping all the duct tape in place. They know every nook and cranny of the app and how it works. They're the glue that holds it all together and are typically so damn busy they often can't be counted on to give you the time and attention you'll need as a QA'er (note...fear not, over time, you may elevate to this position...if you really want it that is...say goodbye to your social life). Important note here...offer whatever help you can to this person, they will be one of your most valuable assets.
The Young Guns - Ah, the energy, the optimism, the talent, the bravado, the inexperience, the arrogance. I'm joking honestly here. Most startups recruit younger staff because it's cost effective and they can get more hours out of them. What they lack in experience they more than make up for in a willingness to experiment. Tap into that eagerness by challenging them to help you out by doing some unit testing, build automation or test harnesses.
Customers - Do you have any? No. Move on. Yes? Congrats. As a QA'er, I can't say that this will be a valuable asset at the beginning, but I'll hedge that with "it depends". You may have one single customer that you're building your app for, if so, make them very happy and involve them in the testing and discovery as much as you can. In my current case, we have a ton of them and a support team to handle the front end. Some day, when we're a grown up QA team, we'll look to engage them more.

What needs my attention the most?

Technically, you need to identify the areas of greatest risk. You'll need to engage the developers and get their input on what areas of the code are 'iffy'. Most developers will admit a hack is a hack. Startup code has lots of them because they build a feature, tweak it, change the feature, tweak it, etc... By the time they're done, it's completely different than what they started with and unless they scrap the code and re-built it, it's probably a mish-mash by now.

Look in the bugdb for counts v. the different components? This can tell you a) what features the end-users are looking at the most (most bugs) and b) where there are no bugs, this might be a newer feature and very likely, untested.

Be very sensitive to "high visibility areas". They may not contain the most complex code, but these are the areas where execs hate to see sloppy bugs. It is amazing what a clean UI can do for an application as the first impression is very, very important. This is also important because these same execs will be demo'ing the wonder-app and if it looks like crap, it likely will be hard to sell. I'm not suggesting that Todd Garland build your site (although he would for the right price), but the on-screen controls should all work correctly and the page layout should be clean and legible.
Wherever the app is doing the heavy lifting (web or data layer), have developers build unit tests that exercise the core code so you can focus on the functional integration and system-level tests.

Phew! Well, that pretty much summarizes your first day at a startup.

See ya on Day #2.

Intro to Insane Startup QA (Part I)

2007-12-13T16:17:00.000-05:00

The Asylum is open.

Welcome to my first post on my new blog. A quick introduction is probably in order I suppose. My name is Drew. I've been working in Software QA for over ten years. Sorry, but if you don't know what QA means, you're probably not going to like this blog that much. I've worked at small, medium and big companies at one point or another and by far, the most enjoyable times I've had have been at startups. Why is that? Don't know for sure, maybe it's the energy, the excitement of being part of something brand new or maybe I guess I'm a glutton for punishment...or an idiot...or, well, insane.

Anyways, this blog will be about my trials and tribulations at my current gig, HubSpot, as well as reflections back on past experiences at other companies, and I'll share stories that I've heard from other people. Basically I'll post about things that interest or amuse me and I'll try to share some of the challenges I face and tell you about how we're attempting to overcome them. I'll be merciless on the guilty, make fun of the non-believers and in general, try to be occasionally funny...oh, and maybe we'll both learn something along the way from one another. It's a very helpful trait to have a good sense of humor if you work in QA at a startup.

Ok, for those of you that have never worked at a startup before, here's the 30,000 ft. view (and yes, I will abuse and re-use old & new buzzwords and terms that I've heard over the years, that's the "net-net" of it my friend):

Startups are companies that don't have a lot of people to do all the work that needs to get done.

Startups typically don't have a lot of money, but they will give you stock options, free food & sodas,and occasional pillow talk if the project is at risk of missing the date they promised.

It is a myth that you have to work a lot of hours at one. Wait, no, that's true, never mind.

Usually they're building their 1.0 product, unless they're a Web 2.0 company, then I'm not too sure what version they start with.

Usually they're wicked smaht people working there (yes, I live in Bahstan)...they have to be smart in order to get all the work done (see first bullet).

There is a lot of pressure typically because of some richer-than-they-need-to-be Venture Capitalist who wants to see a return on their investment before Britney Spears returns to re-hab.

You get to know your co-workers really fast, mainly because they're sitting so close to you that they're practically in your lap or sharing your monitor.

Ok, there ya go, that's a good enough primer on life at a startup...now...how about QA life at a startup you ask?

The unique thing about working in QA at a start-up is that you're typically brought in well after a product has been prototyped, demoed and revisioned about 200 times as they figure out all the whiz-bang features that they're building. There are requirements for this wonder app, they're just in emails that you don't have, whatever open source bug db they installed when they got tired of tracking bugs by email, in the nether-regions of IM-land or they're inside of someone's head. I know what the QA purists are thinking...this is contrary to the mainstream QA battle-cry of "QA should be involved from the very beginning." Well, reality (yes, I said R-E-A-L-I-T-Y) is that if you're starting up your company and you have to "build" something, most QA engineers aren't developers that build applications, so you're going to hire developers instead or you will not be very successful at getting venture funding if you don't have a product...or so I'm told.

Well, by this point they've convinced some VC to throw some money their way and now they can start to hire more people to market and sell the wonder-app and they can actually start to think about things like office space, 401k, better health-care, etc...

So, they're getting closer and closer to wanting to launch this puppy and a few of the sales people they've hired to sell it have said that the wonder-app is really cool, but really hard to demo because of all the issues with it and only one or two people in the whole company actually know how to use the thing end to end.

Uh-oh, it's time to "test some quality into the product" and fast, we need to get some QA in here pronto!

Enter QA.

(to be continued...)