Confessions of a retired bug screener

Confessions of a retired bug screener

My good friend Anika, the ant bear

In the context of this article, “bug screening” means taking a bug report filed by an employee or developer or someone else. Then, doing any number of things:

  • Clean it up to make it more clear and understandable
  • Figure out if it’s already a known issue
  • Try to reproduce the problem
  • Send the bug report back if more information is needed
  • Route the bug to someone
  • Set a priority and perhaps a milestone where the bug should be fixed

Most of my jobs at Apple involved screening bugs and in one 30 month period I screened over 45,000 bugs. Not surprisingly, it was 100% of my job. The reason this is important to mention (other than it gives me a lot of credibility) was that 15 years earlier, the bug flow was small enough that it was only a small fraction of my job.

These tips are a mashup of ideas to be used by bug filers, screeners, engineers, and management.

It’s never too early

Over the years, I saw many new, small teams being formed and I worked for a startup that was small. Very little attention was paid to a strategy for managing bugs. There’s the feeling that it’s just not a big deal right now and we’ll kick the can down the road.

As someone that first started screening bugs for a project that had less than 200 existing bugs to a project 13 years later than had close to 10,000 bugs, a company can grow very fast and you need to get ahead of the curve.

Don’t be the 10 person startup that didn’t think about this until you’re suddenly the 1,000 person startup without the time or personnel to clean up the mess you could have avoided.

Here are some tips I would give to any company setting up a bug tracking system.

Bug titles

Think of the title as the headline for the bug. When you read a headline, you should learn much about what the bug is about.

I change the title of nearly every bug that I screen. Partly it’s for the reasons listed below, but partly it’s because I think tweaking or rewriting the title helps you retain a memory of the bug itself.

  • Titles that read like short sentences are easier to parse when reviewing search results or in bug review meetings. A consistent feel to bug titles often means you know what it’s about without wasting time.
  • Developer a consistent naming convention for technical terms. For example, alert vs notification, upgrade vs update, active vs enabled vs checked, navbar vs button bar. This will help when searching later.
  • Remove metadata from the bug title. For example, configuration info can go into a configuration field. Other stuff is redundant and clutters up the title field and spoils the quick readability of a short title.
  • Use spelled out words rather than abbreviations for readability and use actual product names vs codenames for shipping products

Here’s a (actual, sanitized) before and after bug title:

Before: [FIELD_FC][v13.0][Build: 132.5][CC: 0.22.02][Contacts] – A71 – Adding New Contacts, Not possible to leave the page – “< All Contacts” on Top Left is missing” – NY

After: Back button doesn’t show up after saving new contact

NOTE: This is an actual before and after bug title. All the other metadata was used by the person filing the bug but there were appropriate fields where they could be put to get a human-readable bug title.

Screenshots and Movies

Sometimes I feel like a screenshot or movie should be mandatory for filing a bug report, even if you think it’s obvious what you are talking about when you create the bug report.

As a screener, always check for attached screenshots and movies first, especially if the bug title is unclear. If there is a movie of something that is more easily explained by one or more screenshots, take screenshots of individual frames of the movie and attach as screenshots. This saves a lot of time for people viewing the bugs later, like in bug review meetings.

Keywords

Most bug tracking systems allow keywords to be used, but they can sometimes get out of control. Here are some interesting uses for keywords:

  • Mark high volume bug reports so you can find duplicates easily when new bug reports come in
  • I used to mark some bugs as “easy fix” for less involved bugs. Junior members of the team or interns or those wanting to lend a hand could grab those
  • Sometimes I’d use a keyword to track similar bugs that I wasn’t quite sure were the same. Even if they aren’t the same, the engineer fixing them can more easily fix multiple similar bugs at once.

A variation on this can be employed if you have this capability in your bug tracking system. If you can keep a search results list open that matches a keyword, this can make your life a lot easier. For example, if you get in a new bug and you’re certain you’ve seen it before, check the short list and your job is done very quickly. If It’s not in the list, add the keyword.

Be decisive

Especially early in a development cycle, be decisive as a screener about -setting priorities and milestones. Don’t be timid and afraid to assert your opinion on a bug.

If your bug tracking system allows you to follow bugs, it’s a great way to see if others agree with your assessment if they make changes. You can learn then from your decisions. Nearer the end of the cycle, you’ll be a pro, although it can’t hurt to get a second opinion for bugs where you are on the fence.

Text substitutions

A lot of times you’ll find yourself typing boilerplate text into bugs and it can get tedious. Although there are a lot of 3rd party apps like Alfred that can help here, macOS also has a built in system.

Text substitutions are found in System Preferences > Keyboard > Text. You can enter a string of text (paste it in if you want newlines) and then decide on a shortcut to trigger it. Personally I use short words with an asterisk at the end so I don’t accidentally trigger them.

Some places you can use them:

  • Insert a commonly used search phrase into the search field, including wildcards
  • Insert boilerplate text when sending a bug back for more info from the originator
  • If there’s an enhancement request you don’t plan to do, it doesn’t hurt to have a carefully crafted boilerplate to let people down easy

Unscreened bugs

This seems like a dumb thing to say, but you should constantly monitor your unscreened bug report queue and always keep it as close to zero as possible. There are several good reasons for this:

  • It can get stressful to get behind!
  • Bugs filed may be about code very recently checked in, so if you can get a bug report to someone quickly, they can fix it more quickly
  • If you are about to send out a build, don’t be the guy that missed the super serious bug sitting in their queue
  • Impress your manager
  • A team that is responsive to bugs is one where people are more likely to find other bugs they found.

Bugs bypassing the screener

Sometimes a bug will be assigned directly to an engineer or plucked out of the screener’s queue before they get a chance to screen it. I feel it’s important that a screener is keeping tabs on every bug their team is working on. Bugs that bypass a screener cause a few problems:

  1. The screener would not be aware of any bugs that were not screened by them
  2. They may not have the appropriate milestone, priority, or other information filled out in a standard way
  3. They may actually be duplicates and it’s not great to have those floating around. Imagine two people working on the same bug!

Depending on how sophisticated your bug tracking system is, you could create a query to find these bugs. Here’s how I handled this.

  1. Create a keyword. I used “cricket reviewed”
  2. Every bug assigned to me would get that keyword applied automatically. This may have to be scripted, which would be unfortunate.
  3. Create a query that finds any bug assigned to your component area that does not have that keyword
  4. Assign the bug to you, screen it, get the keyword auto-applied, and send it back to the appropriate person or originator if more information is needed.

Cleaning up the bug tree

  • With any downtime, try to get rid of old bugs in your are. Examples:
    • Look for unreproducible bugs or enhancement requests that are older than a year that have no other reports.
    • Look for bugs that are crashes or hangs where the code in the backtrace no longer exists
    • Bugs that are just really old with no other reports
    • Bugs that haven’t been modified for years

The more cruft you have in your bug trees, the more crap you’ll have in search results, the more time will be wasted sorted through them, and the more time wasted in bug review meetings.

Proactively, try to keep as many new bugs as possible from entering your bug trees until they are “legitimate” bugs. If you feel comfortable enough identifying enhancement requests that are just never going to happen or don’t fit the company ethos, send them back. If they are bugs that involve highly unlikely scenarios that users will never hit, send them back.

Bugs that don’t exist

There are bugs in your software that don’t exist and never will exist. Until they are in your bug tracking system.

At small companies and in small teams, there’s a tendency to report bugs to:

  • Individual engineers
  • Team mailing lists
  • Wide mailing lists
  • Walking to someone’s office

Now there’s nothing wrong with that, but how do you know your bug will be remembered? How can you check the status? How will you know whether it’s been fixed?

When sending to a mailing list, is the right engineer on the list or paying attention?

The general rule is to always file a bug when you encounter something that is obviously a bug, want to suggest an enhancement, or when you have a crash/hang/data loss situation. Even if something just seems wrong but you’re not sure or it confused you, the team will want to know. You don’t want your customers to have a bad experience so file a bug report.

There’s nothing wrong at this point to apply pressure, but file the bug first.

Finding the right team for your bug

Depending on how large your company is, it may be daunting to find the right place to file it.

Here’s a trick to finding the right component:

  • Do a regular search and look for a product name or obvious word in the title field and see what comes back. Chances are many of your results point to where the bug should go or to whom it should be assigned.

If you can’t find a place, encourage your company to have top-level components where you can file bugs if you can’t figure it out. You shouldn’t make people waste time trying to find a component.

Describing a bug

I’m strongly in favor of having very simple templates for reporting bugs. The more onerous you make it, the less likely someone is going to want to file a bug. The key pieces are:

  1. A succinct and descriptive title
  2. Your starting state, steps to reproduce, and what happened
  3. Optionally, what you expected to happen (sometimes, what happened is considered a bug to you and expected behavior to the engineering team)
  4. If something broke and you remember when it worked last (or have the time to test it), that is often invaluable information

Filing duplicate bugs

Don’t waste time trying to figure out if a bug you’re filing is a duplicate or not. It will take you far longer to find a bug in a component you don’t screen for than it will for the screener. Also, duplicates can indicate to the team that a problem is more serious than you thought. Also, you may have a missing piece of the puzzle that will help the bug get fixed.

This is a controversial opinion but was the prevailing opinion at Apple and was advantageous even as bug numbers became extremely high.

Low-tech solutions

My two favorite low tech solutions:

  • Whiteboard
  • Label maker

With a label maker, I can label multiple devices. With a whiteboard I can write down build information. When I did this, I made very simple naming decisions like:

  • T = today’s build
  • Y = yesterday’s build
  • C = current build

Or, if it was a reference build, I might just label it with 1.2.1b5

Then, on my whiteboard, I have a table translating T, Y, and C to build numbers. You could also include the iOS or whatever OS you are using’s version number for each device.

This’ll save you navigating through a bunch of hierarchical menus when filing a bug. Also, if you have a bug in “current build”, grab the device labeled “T” and see if it reproduces. If not, try “Y” or any of the ones labeled with individual builds

Common sense bug filing

  • File a bug as if someone that knows nothing about it could take it and reproduce your bug
  • File bugs that are the type that you would want to receive yourself as a screener

Text editing

Learn the Cocoa text editing system. So incredibly useful and saves mousing and trackpad use. The faster you can type as a screener the faster you can screen bugs. Also applies to bug filers as well.

Conclusion

This list is a jumble because screening bugs for 20 years has a jumbling effect. But had I known all these things back then, I could have saved a huge amount of time and stress trying to keep my head above water.

Comments are closed.