Bug Herding Explained

This document is intended for advanced TSF members. If you just joined us, come back later.
There's plenty of more urgent stuff for you to study, don't read this one yet.

The TSF is made up of volunteers who usually only have a fraction of their time to offer to our cause. They come and go, some of them disappear completely after 500 answers (some after 10, some without even leaving a single one). Our organization must be flexible enough to survive this.

Among other things, this means that our Trello boards must be accessible and easy to use for anyone, even if they are just passing by. Therefore, we need dedicated people who can take care of the boards and keep them well maintained for the rest of us, we need bug herders.

This manual is for those who would like to become part of this caste.

Herder protocol

You know from the bug handling manual that our final goal, when it comes to issues, is to get a trello card marked either as confirmed by dev or feature. As bug herders, we have more detailed responsibilities. We follow issues from the moment they are found to the public release in which they are resolved — and beyond.

  1. Make sure that all newly discovered issues have their own cards (exceptions can be made for problems found in a recent beta version if the dev is in active development and says he'll fix it on the go).
  2. Optimize all newly added cards, they must be: easy to find (good name, search keywords), specific (as in 'not vague'), have all the necessary details, but not too verbose.
  3. Report all issues to the relevant developer group, not forgetting to include the trello link and an #issue_tag (the same way we do in questions).
  4. Find all the requested info and add it to the card, move it to the Reviewed & Reported list, add a comment about this to the card so that we know when this happened.
  5. Remind developers about trending issues that have been neglected for too long.
  6. Check issues that are marked as fixed (are they?) and add relevant comments.
  7. Whenever a new version is out, mark issues as fixed and comment with the version number.
  8. Older issues that were fixed need to be archived or deleted after a while (but not immediately, since we will still be getting reports from people who forgot to update their apps).

Herder goals

A bug herder has nothing left to do, if the Unsorted list is empty. Any card in that list should be seen as a call to action. Now let's look at the job in more detail.

Creating cards

If you found a problem, there is a good chance that somebody else from the TSF will soon get a question about it too. So it's important to create a trello card early. Create the card as soon as you have enough info to set the issue apart from the rest.

Information in a good card is ordered in this manner:
Problem (what) > Conditions (when) > Explanation (why; optional)

The Problem must be described in specific terms (photos are blurry, app crashes, voice messages stop playing).
The Conditions include:

  1. The exact circumstances of the problem (photos are blurry in secret chats, app crashes when opening the contacts section, voice messages stop playing when a new message is received).
  2. Steps to reproduce the issue.
  3. Affected device/OS combinations.

The Explanation part is optional, but it's nice to have it in cards that could potentially live long lives — e.g., when a bug depends on a specific device/OS combination and can't be fixed easily. (For example: 'This happens when photos are sent from an older version of the app and will stop being relevant when everyone updates.')

Title

The name of the card should be pretty general, details should be placed in the description. When composing the title, think from the perspective of somebody looking for the issue, who does not know too many details yet.

A good title should be structured this way:
Problem, general condition

Specific conditions and the explanation go to the description.

E.g., this card is useless: 'If the contact's name begins with “A” or “S”, iPhone 4s (7.1.2) users will not see messages from them'. When somebody encounters this issue, he will not have enough info to find this card. Most likely, he will only know that an iOS user doesn't get messages from some of their contacts. So if we want someone looking at the card to know that it describes their issue, we need to name it like this, for example: 'User doesn't see new messages if they come from contacts with names starting with “A” or “S”'. We will add a description, saying that the problem was encountered in iPhones 4s with iOS 7.1.2. And we will add all the necessary tags to make search easier, e.g. 'iOS, messages, invisible, not delivered', etc.'

Description and more info

Once you've got a card going, gradually expand it: improve the title, optimize keywords for searchability, add more info, don't forget adding #tq tags from affected users to the comments. Try to avoid any vagueness and ambiguity. E.g.:

  • Avoid 'sometimes', 'in some cases', etc if you can. These words signal that additional investigation is required.
  • Use specific terms. Not 'doesn't work', but 'app crashes' or 'becomes unresponsive', etc.
  • Don't use 'latest', 'current' or 'previous' in terms of app version. Better stick to actual version numbers — some cards live long lives.

Generally, you need to strike a balance between enough info — and not too much. Developers are busy people, they will not have time to read through a long text, or might even get scared away by verbosity.

Bad example:


And here's almost the same amount of info. But with higher chances to be read by the developer:

One card — one issue

If you read the examples above, you will have noticed that the initial card was dedicated to two bugs instead of one. This is very bad. Each issue must have a card to itself, even if the problematic behaviour seems similar. Imagine that your house had a leaky roof and burst water pipes at the same time — both problems mean it'll be very wet inside. But the underlying issues are very different.

Keywords & Search

The title and text of the card are searchable, but sometimes we need additional keywords. We usually mention them at the bottom, after a separator. This is useful if there are many ways to refer to a problem.

Once you've created a card, it's always a good idea to try to search for it yourself. Imagine that you've just met a user with this issue and are looking for info on Trello. What will you type in?

  • This may give you ideas for additional keywords.
  • You may find that there are older cards from the 'Fixed' list are shown at the top of the search so that the card is not visible. Check if it's time to delete them.
  • You may find that another card is showing up because of bad wording. Try to rewrite it or remove excessive keywords (but only if they are unnecessary).

If there is a card and you had to show it to one of our volunteers who couldn't find it, think how you can improve its wording and keywords.

E.g., you have a card called 'App freezes when user joins supergroup'. One of the new volunteers asks in their regional group: 'I have a user that says his app crashes when people add him to groups'. You show them your card, but realize that it didn't mention the words 'crash', 'add', or 'group'.
So you add them to the list of keywords below the card. Then you check if it's now possible to find your card using a different description of the problem. You find that a card from 2013 called 'A group of aliens crash-landed on DC1, adding more trouble for the admins' is shown first for this query and delete it because it's a lie and therefore shouldn't be on Trello.

Affected devices

Be careful when mentioning affected devices in the card's title or description. If you can reproduce something on iOS8 — it doesn't mean this is an iOS8 bug yet. This is an iOS8 bug only if you can't reproduce it on iOS7.

Hence, the general rule: unless there are devices where the bug can't be reproduced, don't place device/OS version info in the title. Put them in a list under 'reproduced on'.

As soon as we see some kind of consistency, we can add things to the title. E.g., if we see that all iOS8 devices have the problem and devices running any other version of the OS do not.

Reporting to developers

Bug herders are also specialists in developer relations. Here's what we should do:

  1. Let's stick to the existing app groups for all our contacts with the devs.
  2. Let's contact developers when we already have an issue card going — at least a very basic version. (Exceptions could be made for bugs in beta versions that are in active testing.)
  3. When contacting the devs, describe the issue, add a trello link and an #issue_tag. This way we can later look up what exactly the developer said about this or that issue (or didn't) and act accordingly. (If issues are investigated in local TSF groups, do insert the #issue_tag there to make the relevant info discoverable there as well.)
  4. Once the issue has been optimized and reported, move it to the Reviewed & Reported list.
  5. If the developer asks for more info, supply it to the group (and the card if relevant). If you cannot do that right away for any reason, add the questions as a comment to the card, and add the yellow label.
  6. Remember that developers are biased and will frequently tell you that the problem belongs to some other developer or even is related to the OS or device and is not our bug. If this happens, it is your task to make sure that we've traced the problem back to the right dev (in the former case) and that we can't demostrate other apps that do the same well on the same device or platform (in the latter).

Archiving issues

Once an update is out that fixes an issue, we add a comment along the lines of 'fixed in v.2.9'. Then we leave the issue on the board for about 2 weeks or so — our volunteers will still meet people that need to upgrade to fix their problem. Sometimes an issue deemed to be fixed can also come back to life (likely in a modified form, e.g. 'now only for iPads', etc.).

Once the 2 weeks are out, we can assume that the issue is indeed fixed for good. We can then archive the card. Note that archived cards will still show up in search (unless you specifically exclude them). Therefore, it may be more reasonable to delete the cards in case of smaller bugs and interface glitches (this is especially true for interface oversights and feature-like issues).

It is acceptable to archive an issue that couldn't be reproduced by us and had no activity (complaints, #tq tags, etc.) for 3-4 months. But don't delete those.

Archive:

  • Trending issues
  • Major issues
  • Inter-app issues
  • Hard to reproduce issues
  • Obscure issues without any recent activity

Delete examples:

  • Wrong error message when setting a username starting or ending with an underscore
  • Shared media / shared files selector glitch in landscape mode on iOS 7
  • Can't forward photo when opening from shared media

..and all other minor interface bugs

Deleting cards

Since anything that is deleted is lost forever, we shouldn't rush this. But if you're sure, go ahead and do it. If you're not sure, make sure it should be, then go ahead and delete it.

Herder responsibility

When you create a card or review & report somebody else's card, subscribe to it. And follow it to the end. In the event that something has to be done with it and nobody is doing it — you do it. This way we will not have Mexican standoffs when everybody knows something has to be done, but since technically nobody has to do it, it's never done at all.

And that's about it. If we all observe the simple procedures outlined above, peace and prosperity will come down on our homes and trello boards — and all TSF volunteers will only be a few keystrokes aways from any answer they might need.

How do I join?

You talk to Markus or Daria. But bear in mind that he'll like to see some 20-30 bugs you've created, reviewed and otherwise took care of on Trello. All bug herders should be prepared to spend a lot of time answering questions and tending their issues.

P.S. Bugging the devs — a note on bug relevance

Bugs and even trending bugs have different priorities. Those priorities can be calculated in the same way as one would determine whether a feature is worth considering.

I mentioned 'reminding the developers about bugs' as a responsibility of the herder, especially of a herder who is subscribed to a bug. But do remember that there are good and bad times to do this.

Whenever an update is imminent, it is a good idea to go through existing issues and make sure that all the critical stuff is fixed. Don't bug pre-release devs with exotic stuff — if it wasn't fixed at this point, it'll have to wait until the next release.

The best time for reminding about older issues would probably be a few days after a major release, or as soon as the dust has settled — maybe one day after the release if no major new bugs were found.

Such a reminder is best done in the form of a digest. You could collect a few links to well-investigated issues and offer them for consideration in the app's group. Alternatively, you could also gather a few issues that are less clear and ask about them again. Just don't do it in 20 separate messages over the course of seven hours — make it one nice but not too fat message.

If that doesn't work, try a smaller number. Maybe choose three important issues and ping him again with a nice concise message. Once he says something of that is fixed, show him your earlier digest and ask to take care of that stuff as well since he's here already.

Be cool.