who is starring Firefox tinderbox failures and how long before they do?

Using the accumulated data in my CouchDB instance about the Firefox tinderbox and Protovis, we can see how long it takes before a testfailed/busted build gets starred and who stars it.  The horizontal axis is the number of minutes between the end of the build and when it gets starred.  If someone stars the build before it finishes running, a negative number is possible.  The vertical axis is a double-height histogram, which is to say the scaling is percentage based and don’t try to infer absolute numbers.  Colors tell us who did the starring.

The winner is philor.  I knew that going in, I just wanted to see it.  Thank you for reducing the pain of dealing with the tree!

more systemtap mozilla event loop awareness

====================                                             ms    #
----- Event Loop:
  nsInputStreamReadyEvent                                      7119   11
----- Input Events:
  nsInputStreamPump                                            7119   11
----- Pump Events:
  thunk:nsNNTPProtocol::QueryInterface(...)                    7119   11

====================                                             ms    #
----- Event Loop:
  nsInputStreamReadyEvent                                      6284   15
  nsTimerEvent                                                  242  123
----- Timers:
  nsUITimerCallback                                             121    2
----- Input Events:
  nsInputStreamPump                                            6284   15
----- Pump Events:
  thunk:nsNNTPProtocol::QueryInterface(...)                    6283   15

As of late, I’ve noticed that Thunderbird has been pegging one of my CPU cores periodically in ways not correlated with gloda activity, the usual scapegoat.  With some enhancements to my systemtap script from last time, I can now see that it apparently takes something like 13 seconds of dominating the event loop to perform the check for new messages in my 26 subscribed news folders on news.mozilla.org.  Yowch.  At least this is on a debug build!

Changes are that we now cover input stream ready events, input pump listeners, and proxied objects (which lets us see what Thunderbird IMAP gets up to).  Also, the output is more concise and only shows things that take at least 100ms of aggregate wall clock time.

Script invocation currently wants to look like this:
sudo stap -v moz-event-loop.stp /path/to/thunderbird/objdir/mozilla/dist/lib/libxpcom_core.so /path/to/thunderbird/objdir/mozilla/dist/bin/components/libnecko.so | ../addrsymfilt.py `pgrep thunderbird-bin`

Thunderbird ĝLȬdÅ full text search tokenizer now supports accent folding, non-ASCII case-folding, and more!

Thanks to the efforts of Makoto Kato (jp-blog en-blog twitter) whom you may remember as the bringer of CJK bi-gram tokenization, I have just pushed tokenizer support for case-folding, accent-folding, and voiced sound mark magic to comm-central.  The net result is that searches for “ĝLȬdÅ” will now find “gloda” and vice versa.  Less crazy searches should also be improved.

Starting tomorrow, Thunderbird (in the guise of Lanikai) 3.1 nightlies will have this capability and it will also show up in the next beta (beta 2).  No action on your part is required; when you start running a build with this capability your existing gloda database will get blown away and indexing will start from scratch.  If you go back to an older version for some reason after having used the updated build, the old build will also decide to blow away your database, so try to avoid making a habit of switching between them.

We also have some significant performance enhancements related to gloda search and indexing slated for beta 2, not to mention* usability enhancements to avoid the dreaded gloda search/quick search toggle dance.

* This expression is weird given that I go on to mention the allegedly unmentioned, but it still makes more sense than “all but X”.

partial: posting pymake data/results to a couch

What it does:

  • Adds a “–couch-json” option to bsmedberg‘s pymake that takes a URL that crams the state of (global, non-rule associated) variables into a couch document for each makefile context executed.  Variables are grouped by general origin as well as specific Makefile origin where relevant.  The code as it stands only tells you about make variables that were accessed because this cuts down on things you (I) don’t care about.  (For example, if FOO and BAR are defined but only FOO is ever expanded, then only FOO goes in the couch document.)

I think there’s a fair bit of potential in this (pushing “make -p” style data/make output into couch), at least in situations where there’s no escaping make, you don’t already have usable/appropriate tooling, and python/CouchDB/JSON is one of your religions.

I’m hoping to avoid touching the mozilla build system in any meaningful fashion from here on out, so I am unlikely to work on this much more, but it could be useful to others, hence this post.  The repo is here, check out the “understand” (p)branch.  (The changes are not suitable for upstreaming as things stand.)

Tinderbox results in bugzilla, jetpack times 2, CouchDB, review board

mstange‘s Tinderboxpushlog is awesome.  You know it, I know it, the many people whose sanity has been saved by it know it.  It is a fantastic improvement on checking the tinderbox; it lets you know the current state of the tree, the recent history of the tree, and how these things are correlated with recent commits.  What it is not good at (nor intended for) is to be a historical record.  Tinderboxpushlog ‘scrapes’ the tinderbox on-demand using tinderbox time windows and does not have the ability to key off anything but the time.

So if one refactored the scraper into a CommonJS module suitable for use with the newly rebooted jetpack, hooked it up to a cron job, and crammed its outputinto a CouchDB database, what would you get?

Exactly, something you can hook up to johnath and ehsan‘s magic bugzilla jetpack (last mentioned on the blog here).  You can install it from here.  (johnath/ehsan, please feel free to pull the changes into the upstream repo; I’m still wary of randomly pushing things into other people’s user repos…).  I know the presentation is ugly, not the least because the text labels are inconsistent with tinderboxpushlog; feel free to push into my user repo with improvements.  Oh, and for this to work you need to put an honest-to-goodness URL in your bugzilla comment or attachment description; there is no all-powerful regex if you type things out by hand.

You can find the thing that pushes thing into the couch using refactored tinderboxpushlog logic here.  Some cuddlefish runner tweaks (not all of which are likely advisable) can be found in my user jetpack-sdk repo.  (The new jetpack reboot is wicked awesome, by the way.)

Right now the cron job is running against the MozillaTry, Firefox, and Thunderbird3.1 trees on the tinderbox every 5 minutes.  While it should be pretty easy for me to keep the cron job and couch server online, I make no guarantees about the schema used in the couch, just that I will keep the jetpack in sync with it.  And if the service starts exceeding the resources of my (personal) linode, I may have to tweak polling rates or what not (‘what not’ meaning ‘up to and including turning it off’).

There is other work happening in this area and I am excited about that.  For example, I think brasstacks has an encompassing vision that should help provide historical data about which tests failed, etc.  With any luck, my efforts here will be mooted by buildbot and magic awesome dust.

The mention of reviewboard in the title is just that if you are using my review board stuff and put a link to your try server commit in the attachment description, we will use that to pull the official hg changeset as the basis for the diff.  The main benefit of this is that if your patch depends on other patches in your queue that are not yet in the trunk, the diff will still work out.  Specifically, if your queue had A, B, and C applied (where C is the tip) and you link to C, then we will provide a diff of C relative to B.  Please be aware that hg.mozilla.org is rather slow about providing the hg changeset diffs on demand so this will be at least an order of magnitude slower than the fetch of an already-available patch from bugzilla.  Repo with changes is here.

Thunderbird Message Filter Bar Prototype Extension, check it.

What was just the quick-search box in Thunderbird 2 is now also the home to global search in Thunderbird 3.  This hasn’t turned out splendidly, although we didn’t expect that it would.  Some people think quick-search is gone because they do not realize you can change the modes of the search box.  Other people are hardcore and know how to switch between the modes, they just don’t like all the clicking.

So we’re thinking about splitting the quick-search out into its own separate box.  In addition, we’re trying to expose a lot of the power of the “mail views” system.  You may know “mail views” as that boxy thing that lived with the quick-search box above the thread pane in Thunderbird 1.5 but then became something you had to customize onto the toolbar at some point.  It looks like this:

Thunderbird has a very nice search subsystem under the hood that powers quick-search, “advanced search”, virtual folders, etc.  Mail views was and is the mechanism that allowed you to define arbitrary searches and use them as filters on any folder.  Unfortunately it’s not a smooth operator and its defaults have some issues.  There’s no “starred” filter unless you define your own, there’s no “any tag” filter unless you define your own, and “People I Know” only checks one of the two address books you are likely to have in your profile.  Even with those defined, you’re looking at 3 clicks to get to most things.

So the message filter bar is also trying to bring one-click access to these things you might care about.  In the top screenshot, that’s what you’re looking at.  Starred messages, messages from people in any of your local address books, messages with tags, and messages with attachments are all at your fingertips.  The new quick-search location over on the right works right with them.  And if you love mail views and all the clicking-finger muscle strength it helps build, it works with mail views too!

One bit of polish that I’m hoping people like and performs sufficiently well is the tags case.  When you haven’t clicked on the tags icon, the bar in the screenshot does not show that bit with the tags.  When you click on it, it 1) filters the visible messages to messages with tags, and 2) figures out what tags are on those messages and populates that bar with those tags.  You can then click on any of the tags to stop including messages with that tag (and none of the other tags still selected).

In any event, if you are interested, the prototype is being developed as an extension that you can find here on AMO (sandboxed) if you like pre-built XPIs and here in hg if you like source code.  It is very prototype-y at the current moment.  Keybindings aren’t there, localization/accessibility is not there, being able to make the filter bar go away isn’t there, etc.  We will iterate on things and productize assuming the concept works out.  Just be aware that I don’t believe sandboxed plugins auto-update, so you if you’re really interested you might need to keep an eye on the AMO page or the repo.

UPDATE: I have nuked the add-on from AMO since Thunderbird 3.1 beta 2 is now available.

Review Board and Bugzilla reviews, take 3

I’ve updated my review board setup once more (part 2, part 1).  The low barrier to entry is now even lower.  “How low?”, you might ask.  “On the ground!”, I might say.  “What other low low price features with big big value are on offer? With more facts and less spiel?”, you might then also ask…

  • It now works with patches that have a header.  Patches that were the result of an mq import and then directly uploaded will tend to have headers.  This was why sometimes patches would fail to import with unlikely errors about empty patches.
  • There is now a magic URL scheme that automatically pulls the patch, creates a review and bounces you to the review.  If the review already exists, it just directly bounces you.  That URL scheme is http://reviews.visophyte.org/r/bzpatch/bug###/attach###/.  There are no authentication requirements on using this URL nor viewing the diff or associated reviews.  However, if you want to actually use the review mechanism to make comments then you will need to login via OpenID.  See part 2 for more on that.
  • I modified johnath and Ehsan Akhgari’s magic bugzilla jetpack so that it also adds a “Review” link to patches.  My modified repo is here.  You can install it from here.  You can see what the word “Review” looks like up in the first screenshot.
  • I upgraded to the reviewboard git trunk.  This adds some improvements to the diff display such as showing you function context information, even if the patch did not include it.  (And even if it did, too.  A lot of patches involving C++ truncate the function signature, whereas as you can see in the screenshot, you get the full text!)  I believe it is also supposed to be clever about recognizing moved blocks.

Limitations and other notes:

  • Patch fetching is synchronous and can take a while because we also parse it all up before we return.  Do not sit there hitting reload.  We’ll give you an error message if it doesn’t work out.  Not a great one, but an error message nonetheless.
  • Patches are still assumed to be against mozilla-central or comm-central (depending on the bugzilla product) trunk as of the moment we fetch the patch.  This means bit-rotted patches that you are only looking at now may fail to apply.  At the same time, patches where you clicked on the link back when it was timely and go back to look at them now that they are going out of style are still going to be applied against the same revision they were in the first place.
  • The repo with my modified changes (on the bzreview-master branch) was nuked and re-created because of the svn -> git transition by the reviewboard people.  So if you previously pulled, you should probably blow your old repo away rather than end up with a weird hybrid mixture.  (btw, hg-git works very nicely, with the caveat that its bookmark-based representation of the git branching idiom confuses pbranch really quite badly.)
  • Ping me on IRC or drop me an e-mail if you are experiencing reliable problems after having determined that the patch in question is not just full of gibberish.

Using systemtap to figure what your mozilla app’s event loop is up to

====================                                             ms    #
----- Event Loop:
  nsTimerEvent                                                 1233   31
  nsProxyObjectCallInfo                                          19   44
  nsStreamCopierOB                                                1   48
  nsStreamCopierIB                                                0   18
  nsProxyCallCompletedEvent                                       0   44
  nsProxyReleaseEvent                                             0   27
  nsTransportStatusEvent                                          0   19
  nsSocketEvent                                                   0   18
  nsHttpConnectionMgr::nsConnEvent                                0    1
----- Timers:
  OnBiffTimer(...)                                             1129    3
  nsGlobalWindow::TimerCallback(...)                             70   10
  nsAutoSyncManager::TimerCallback(...)                          29    6
  nsExpirationTracker::TimerCallback(...)                         1    1
  nsIdleService::IdleTimerCallback(...)                           0    5
  nsExpirationTracker::TimerCallback(...)                         0    1
  nsHttpHandler                                                   0    1
  nsUITimerCallback                                               0    2
  imgContainer::sDiscardTimerCallback(...)                        0    1
  nsExpirationTracker::TimerCallback(...)                         0    1

That’s one of the periodic outputs (10 seconds currently) of this systemtap script filtered through this python script to translate addresses to useful symbol names in realtime.  It’s like top for mozilla.

Actual invocation looks like so:
sudo stap -v moz-event-loop.stp /path/to/thunderbird/objdir/mozilla/dist/lib/libxpcom_core.so | ../addrsymfilt.py `pgrep thunderbird-bin`

The giant caveat (and giant hooray for utrace and Fedora having kernels with the utrace patches built-in) is that the magic in here is done dynamically using utrace source line probes.  As such, the probes aren’t resilient in the face of changes to the underlying source files; the current line numbers are targeted at 1.9.2.  There are various in-tree and out-of-tree solutions possible.

Thunderbird Jetpack Teasers: Words per Minute in Compose

jetpack.future.import("thunderbird.compose");
jetpack.thunderbird.compose.appendComposePanel({
  onReady: function (panel, composeContext) {
    let doc = panel.contentDocument;
    let msgNode = $("<span />", doc.body).appendTo(doc.body);
 
    let started = Date.now();
    setInterval(function() {
      let words = composeContext.getPlaintextContents().split(/\s+/);
      let secs = Math.ceil((Date.now() - started) / 1000);
      let wordsPerMinute = Math.floor((words.length * 60) / secs);
      msgNode.text(wordsPerMinute + " words per minute.");
    }, 1000);
 
    panel.show();
  },
  html: <><body style="overflow: hidden"></body></>
});

thunderbird-jetpack-words-per-minute-example