Category Archives: Mozilla

Mozilla related things, most likely Thunderbird.

prototype unified JavaScript/C++ back-traces for Mozilla in (archer) gdb

Posted on by

fused-js-cpp-backtrace-2-upper-half

As far as I know (and ignoring my previous efforts on chroniquery along these lines), up until now you had your C/C++ Mozilla backtraces via gdb (chocolate) and your JS backtraces via “call DumpJSStack()” or the debugger keyword from within JS (peanut butter), but these two great flavors had never come together to make a lot of money for dentists.

The screenshots (which is actually just one screenshot split in two) show invocation of a custom python gdb command building on my previous exciting pretty gdb commands.  The command has filtered out boring JS interpreter / XPConnect code and interleaved exciting interesting JS stack frames.

The implementation is reasonably simple and intended to be able to be implemented using VProbes to support my recent performance work along those lines.  We walk stack frames the usual way.  Ahead of time, we have marked out the PC ranges of interesting JS interpreter functions (js_Interpret and js_Execute).  If the stack frame’s instruction pointer is in one of those functions we grab the JSContext argument.  We pop frames until we reach the native frame those functions allocate from their own stack space (whose boundaries we know from the stack walking).

There is one trick we have to do involving dormantFrameChain.  While js_Execute has a consistent and straightforward usage of JS_SaveFrameChain, XPConnect and its quickstub friends are more complex.  Right now we use a dumb heuristic that just looks if our frame pointer is 0 and there is a dormantFrameChain, and in that case we restore it.  (Thankfully the garbage collector needs to know about the shelved frames, otherwise we might have to chase frames down.)  I haven’t put much effort into thinking about it, but the heurstic seems a bit reckless.  We could likely just concurrently walk the XPConnect context stack to figure out when to restore dormant frame chains.  The existing VProbe JS stack (only) code already goes to the horrible effort to get at the thread-local stack, so it wouldn’t be too much more work.  Things probably also fall down during garbage collection right now.

Hg repository is here.  Under no circumstances try to use this with jblandy’s excellent archer-mozilla JS magic right now.  The current code is very distrustful of gdb.Value in a dumb way and does exceedingly dangerous things wherein pointers are bounced to strings and back to integers because direct integer coercion is forbidden.  With pretty printers installed this is likely to break.  Also, this is all only tested on 1.9.1.

fused-js-cpp-backtrace-2-lower-half

So’s your facet: Faceted global search for Mozilla Thunderbird

Posted on by

faceting-gloda-hover-davida-1

Following in the footsteps of the MIT SIMILE project’s Exhibit tool (originally authored by David Huynh) and Thunderbird Seek extension (again by David Huynh), we are hoping to land faceted global search for Thunderbird 3.0 (a la gloda) in beta 4.

I think it’s important to point out how ridiculously awesome the Seek extension is.  It is the only example of faceted browsing or search in an e-mail client that I am aware of.  (Note: I have to assume there are some research e-mail clients out there with faceting, but I haven’t seen them.)  Given the data model available to extensions in Thunderbird 2.0 and the idiosyncratic architecture of the UI code in 2.0, it’s not only a feature marvel but also a technical marvel.

Unfortunately, there was only so much Seek could do before it hit a wall given the limitations it had to work with.  Thunderbird 2.0’s per-folder indices are just that, per-folder.  They also require (fast) O(n) search on any attribute other than their unique key.  Although Seek populated an in-memory index for each folder, it was faced with having to implement its own global indexer and persistent database.

Gloda is now at a point where a global database should no longer be the limiting factor for extensions, or the core Thunderbird experience…

faceting-gloda-action-tag-hover-bienvenu-1

The screenshots are of a fulltext search for “gloda” in my message store.  The first screenshot is without any facets applied and me hovering over one of David Ascher’s e-mail address.  The second is after having selected the “!action” tag and hovering over one of David Bienvenu’s e-mail address.  Gloda has a concept of contact aggregation of identities but owing to a want of UI for this in the address-book right now, it doesn’t happen.  We do not yet coalesce (approximately) duplicate messages, which explains any apparent duplicates you see.

The current state of things is a result of development effort by myself and David Ascher with design input from Bryan Clark and Andreas Nilsson (with hopefully much more to come soon :).  Although we aren’t using much code from our previous exptoolbar efforts, a lot of the thinking is based on the work David, Bryan, and myself did on that.  Much thanks to Kent James, Siddharth Agarwal, and David Bienvenu for their recent and ongoing improvements to the gloda (and mailnews) back-end which help make this hopefully compelling UI feature actually usable through efficient and comprehensive indexing that does not make you want to throw your computer through a window.

If you use linux or OS X, I just linked you to try server builds.  The windows try server was sadly on fire and so couldn’t attend the build party.  The bug tracking the enhancement is bug 474711 and has repository info if you want to spin your own build.  New try server builds will also be noted there.  Please keep in mind that this is an in-progress development effort; it is not finished, there are bugs.  Accordingly, please direct any feedback/discussion to the dev-apps-thunderbird list / newsgroup rather than the bug.  Please beware that increases in awesomeness require that your gloda database be automatically blown away if you try the new version.  And first you have to turn gloda on if you have not already.

Using VMWare Record/Replay and VProbes for low time-distortion performance profiling

Posted on by

profile-performance-graph-enumerateProps

The greatest problem with performance profiling is getting as much information as possible while affecting the results as little as possible.  For my work on pecobro I used mozilla’s JavaScript DTrace probes.  Because the probes are limited to notifications of all function invocations/returns with no discretion and there is no support for JS backtraces, the impact on performance was heavy.  Although I have never seriously entertained using chronicle-recorder (via chroniquery) for performance investigations, it is a phenomenal tool and it would be fantastic if it were usable for this purpose.

VMware introduced with Workstation 6/6.5 the ability to efficiently record VM execution by recording the non-deterministic parts of VM execution.  When you hit the record button it takes a snapshot and then does its thing.  For a 2 minute execution trace where Thunderbird is started up and gloda starts indexing and adaptively targets for 80% cpu usage, I have a 1G memory snapshot (the amount of memory allocated to the VM), a 57M vmlog file, and a 28M vmsn file.  There is also and a 40M disk delta file (against the disk snapshot), but I presume that’s a side effect of the execution rather than a component of it.

The record/replay functionality is the key to being able to analyze performance while minimizing the distortion of the data-gathering mechanisms.  There are apparently a lot of other solutions in the pipeline, many of them open source.  VMware peeps apparently also created a record/replay-ish mechanism for valgrind, valgrind-rr, which roc has thought about leveraging for chronicle-recorder.  I have also heard of Xen solutions to the problem, but am not currently aware of any usable solutions today.  And of course, there are many precursors to VMware’s work, but this blog post is not a literature survey.

There are 3 ways to get data out of a VM under replay, only 2 of which are usable for my purposes.

  1. Use gdb/the gdb remote target protocol.  The VMware server opens up a port that you can attach to.  The server has some built-in support to understand linux processes if you spoon feed it some critical offsets.  Once you do that, “info threads” lists every process in the image as a thread which you can attach to.  If you do the dance right, gdb provides perfect back-traces and you can set breakpoints and generally do your thing.  You can even rewind execution if you want, but since that means restoring state at the last checkpoint and running execution forward until it reaches the right spot, it’s not cheap.  In contrast, chronicle-recorder can run (process) time backwards, albeit at a steep initial cost.
  2. Use VProbes.  Using a common analogy, dtrace is like a domesticated assassin black bear that comes from the factory understanding English and knowing how to get you a beer from the fridge as well as off your enemies.  VProbes, in contrast, is a grizzly bear that speaks no English.  Assuming you can convince it to go after your enemies, it will completely demolish them.  And you can probably teach it to get you a beer too, it just takes a lot more effort.
  3. Use VAssert.  Just like asserts only happen in debug builds, VAsserts only happen during replay (but not during recording).  Except for the requirement that you think ahead to VAssert-enable your code, it’s awesome because, like static dtrace probes, you can use your code that already understands your code rather than trying to wail on things from outside using gdb or the like.  This one was not an option because it is Windows only as of WS 6.5.  (And Windows was not an option because building mozilla in a VM is ever so slow, and, let’s face it, I’m a linux kind of guy.  At least until someone buys me a solid gold house and a rocket car.)

profile-performance-graph-callbackDriver-doubleClicked

My first step in this direction has been using a combination of #1 and #2 to get javascript backtraces using a timer-interval probe.  The probe roughly does the following:

  • Get a pointer to the current linux kernel task_struct:
    • Assume we are uniprocessor and retrieve the value of x86_hw_tss.sp0 from the TSS struct for the first processor.
    • Now that we know the per-task kernel stack pointer, we can find a pointer to the task_struct at the base of the page.
  • Check if the name of our task is “thunderbird-bin” and bail if it is not.
  • Pull the current timestamp from the linux kernel maintained xtime.  Ideally we could use VProbe’s getsystemtime function, but it doesn’t seem to work and/or is not well defined.  Our goal is to have a reliable indicator of what the real time is at this stage in the execution, because with a rapidly polling probe our execution will obviously be slower than realtime.  xtime is pretty good for this, but ticks at 10ms out of box (Ubuntu 9.04 i386 VM-targeted build), which is a rather limited granularity.  Presumably we can increase its tick rate, but not without some additional (though probably acceptable) time distortion.
  • Perform a JS stack dump:
    • Get XPConnect’s context for the thread.
      • Using information from gdb on where XPCPerThreadData::gTLSIndex is, load the tls slot.  (We could also just directly retrieve the tls slot from gdb.)
      • Get the NSPR thread private data for that TLS slot.
        • Using information from gdb on where pt_book is located, get the pthread_key for NSPR’s per-thread data.
        • Using the current task_struct from earlier, get the value of the GS segment register by looking into tls0_base and un-scrambling it from its hardware-specific configuration.
        • Use the pthread_key and GS to traverse the pthread structure and then the NSPR structure…
      • Find the last XPCJSContextInfo in the nsTArray in the XPCJSContextStack.
    • Pull the JSContext out, then get its JSStackFrame.
    • Recursively walk the frames (no iteration), manually/recursively (ugh) “converting” the 16-bit characters into 8-bit strings through violent truncation and dubious use of sprintf.

The obvious-ish limitation is that by relying on XPConnect’s understanding of the JS stack, we miss out on the most specific pure interpreter stack frames at any given time.  This is mitigated by the fact that XPConnect is like air to the Thunderbird code-base and that we still have the functions higher up the call stack.  This can also presumably be addressed by detecting when we are in the interpreter code and poking around.  It’s been a while since I’ve been in that part of SpiderMonkey’s guts… there may be complications with fast natives that could require clever stack work.

This blog post is getting rather long, so let’s just tie this off and say that I have extended doccelerator to be able to parse the trace files, spitting the output into its own CouchDB database.  Then doccelerator is able to expose that data via Kyle Scholz‘s JSViz in an interactive force-directed graph that is related back to the documentation data.  The second screenshot demonstrates that double-clicking on the (blue) node that is the source of the tooltip brings up our documentation on GlodaIndexer.callbackDriver.  doccelerator hg repo.  vprobe emmett script in hg repo.

See a live demo here.  It will eat your cpu although it will eventually back off once it feels that layout has converged.  You should be able to drag nodes around.  You should also be able to double-click on nodes and have the documentation for that function be shown *if it is available*.  We have no mapping for native frames or XBL stuff at this time.  Depending on what other browsers do when they see JS 1.8 code, it may not work in non-Firefox browsers.  (If they ignore the 1.8 file, all should be well.)  I will ideally fix that soon by adding an explicit extension mechanism.

Thunderbird Jetpack messageDisplay.overrideMessageDisplay fun.

Posted on by

jetpack-twitter-follow-notification

As part of our goal to make it easy to write extensions for Thunderbird 3, we’ve been working on getting Jetpack running under Thunderbird and exposing Thunderbird-specific points. This is all experimental, but it’s having good results.

The first example replaces the message you get from twitter when someone follows you and instead shows you that person’s twitter page so you can see what they’ve written. Unfortunately, if you try and click on links on the page you will become sad because they all try and trigger your web browser. But Standard8 is hard at work resolving the content display issues. Besides demonstrating registration via a regex over the sender’s e-mail address, it also shows us extracting message headers from the message. Also, we introduce a small HTML snippet that precedes the nested web browser so it’s not just an embedded web browser.

jetpack.future.import("thunderbird.messageDisplay");
jetpack.thunderbird.messageDisplay.overrideMessageDisplay({
  match: {
    fromAddress: /twitter-follow-[^@]+@postmaster.twitter.com/
  },
  onDisplay: function(aGlodaMsg, aMimeMsg) {
    let desc = aMimeMsg.get("X-Twittersendername", "some anonymous jerk") +
      " has followed you on Twitter.  Check out their twitter page below.";
    return {
      beforeHtml:
        <>
          <div style="background-color: black; color: white; padding: 3px; margin: 3px; -moz-border-radius: 3px;">
            {desc}
          </div>
        </>
      url: "http://twitter.com/" + aMimeMsg.get("X-Twittersenderscreenname")
    };
  }
});

jetpack-amazon-big-total

Our second example of the extension point replaces e-mails from Amazon about an order (order confirmation and shipment confirmation) with the amount of money you spent on the order in BIG LETTERS (or rather BIG NUMBERS). It uses a regular expression run against the message body to find the total order cost. Then it generates a simple web page to present the information to you.

jetpack.future.import("thunderbird.messageDisplay");
jetpack.thunderbird.messageDisplay.overrideMessageDisplay({
  match: {
    fromAddress: /(?:auto-confirm|ship-confirm)@amazon.(?:com|ca)/
  },
  _totalRe: /Total(?: for this Order)?:[^$]+\$\s*(\d+\.\d{2})/,
  onDisplay: function(aGlodaMsg, aMimeMsg, aMsgHdr) {
    let bodyText = aMimeMsg.coerceBodyToPlaintext(aMsgHdr.folder);
    let match = this._totalRe.exec(bodyText);
    let total = match ? match[1] : "hard to say";
    return {
      html:
      <>
        <style><![CDATA[
          body { background-color: #ffffff; }
          .amount { font-size: 800%; }
        ]]></style>
        <body>
          you spent... <span class="amount">${total}</span>
        </body>
      </>
    };
  }
});

The modified version of Jetpack can be found here on the “thunderbird” branch. “about:jetpack” can be triggered from the “Tools” menu. Besides the development jetpack, you can also add jetpacks from the about:jetpack “Installed Features” tab by providing a URL directly to the javascript file. Unfortunately, I just tried installed more than one Feature at the same time and that fell down. I’m unclear if that’s a Thunderbird content issue, a problem with my changes, or a problem in Jetpack/Ubiquity that may go away when I update the branch.

Review Board and Bugzilla reviews, take 2.

Posted on by

review-board-diff

Last time I played with Review Board and bugzilla request queues things were great for me, but no one else.  I had to create an account for you on the instance, add you to my script that synchronizes request queues, run the script, and then keep running the script periodically.  Not to mention there wasn’t really a way to get your review out and into bugzilla.  No one actually tried to use it, so they probably also didn’t notice there were caching issues related to the ever-changing definition of “HEAD” (“tip”).  It sucked, and when I upgraded and everything broke, no one cared, not even me.

But now it’s back and better than ever:

review-board-bugzilla-queue-sync

  • People don’t need to login at all to see review requests and reviews!  Just point them at the URL and away they go.  (Actually, this was the case before, but it was not obvious.)
  • You can/must sign in with Open ID!  If you have Weave and are reading this in the future, Weave can be your Open ID friend!  If you are like me and live in the present (Weave 0.3.2), something is wrong and it doesn’t work, not to mention that Weave takes over the Open ID box so you can’t use credentials that work.
  • There’s a button that updates your request queue for the 16 most recent requests made of you.  Just make sure that you have entered your bugzilla e-mail address on the “my account” page.  This may have happened automatically depending on what your Open ID provider provided/you told them to provide.
  • There’s export functionality so you can take your review and cram it in bugzilla.
  • The definition of “tip” gets nailed down when the review request gets created, so no more ugly caching issues.  Patches can still fail to apply, though, if “tip” has drifted from when the patch was first created.
  • It has friendlier assumptions about what repo you are dealing with.  Thunderbird/MailNews Core/Calendar/SeaMonkey are all assumed to be in comm-central, everyone else is assumed to be in mozilla-central.  Patches against other repos (including mozilla 1.9.1) clearly will not work without additional logic (and some kind of extra info, like people putting “1.9.1” in their attachment descriptions.)

Here is a (fake) example of the “pretty” review output that is possible if you tell people about your reviewboard review (see it live here).  Although it says Bienvenu, it’s just me pretending to be him because his review queue is more interesting than mine.  The comments are accordingly mine.

review-board-review-pretty

Now, what does the export look like (see it live here)?

on file: mailnews/base/src/nsMessenger.cpp line 635
>     // if we have a mailbox:// url that points to an .eml file, we have to read
>     // the file size as well

what a pretty comment

on file: mailnews/base/src/nsMessenger.cpp line 642
>     NS_ENSURE_SUCCESS(rv, rv);

please rename rv ARRRRRR-VEEEEEE

Yeah, it looks like that.

A quick feature summary that explains why this is better than just looking at diffs in bugzilla:

  • Syntax highlighting!
  • It actually has the full-context of the rest of the file!  No more being limited to the 3 or 8 lines of context in the diff you are provided.  I know I have done a lazy review and let a bug through that would not have happened if I had more context at my fingertips (or was not sometimes lazy).
  • People just trimming down your patch to what they are commenting on leaves you with no context of what changed at all!

Useful links:

  • A more interesting live diff to check out.
  • The root of the review board that will prompt you to login via an Open ID account.  When syncing your review queue, please keep in mind that it can take a bit to do all the legwork and you won’t see any feedback until it is actually done doing everything else.  You should get some form of feedback no matter what happens, so don’t keep hitting refresh.
  • The hg repo for my modified version of review-board.  It’s based on an extremely shallow hgsvn checkout.  My questionable development strategy was to make changes with emacs locally, commit, then push to my VM, so the changesets are sometimes a bit excessive.

Caveat usor:

  • This is running on a linode VM right now.  This is better than my local box or what not, but it’s not Mo[MC]o IT or anything.
  • My changes, at least the export functionality, may be buggy.  You may need to rely on me to fix the export functionality to get your stuff out that way.  (If the hg diff doesn’t apply cleanly, you can’t enter data to lose it, so I’m less concerned about that.)
  • I have no plans to blow away the database, but at the same time, please be prepared for the possibility that space ninjas destroy your data.  Use the export functionality and save it to a text file or something periodically if you’re doing a major review.  (In case it’s not obvious, the export functionality is the text labeled “bugzilla-style export” to the right of the reviewer’s name at the top-left of each review.)  You can do your review in multiple passes, exporting each review pass individually.
  • I am confident something will go wrong.  Feel free to post comments here or ping me on IRC (:asuth).
  • If people actually try using this, I’ll stop developing on the live server, but do be aware that apache restarts (lasting a few seconds) may periodically happen, but this should not really impact anything.

Props:

Using BugXhibit to find that bug you know you saw recently but can’t find

Posted on by

bugxhibit-cc-search

BugXhibit, the Bugzilla search results viewer made with the SIMILE Exhibit widget, is now more fancy, and now addresses another one of my use cases.  I frequently find myself wanting to point someone at a bug, or go back to a bug that I know passed through my bugmail recently, and have trouble finding it.  So now BugXhibit can do easy searches based on reporter/assignee/cc/commenter with time ranges.

Examples by way of live links this time (noting that the default time interval is 7 days).  Uh, and if it gives you an error for reasons I don’t fully understand if you open it in a new tab (in the background) from here, just hitting enter in the address bar should fix it.  I’m going to lazyweb that problem for now.

Other changes:

  • It now is also self documenting, just click on “Show Docs” on the page.
  • You can now use arguments to specify the sort and whether grouping is active on the page.
  • The date parsing is better.  Bugzilla doesn’t provide the raw dates but attempts to change things based on how recent the date is.  BugXhibit does a good job of fixing up the date if you are in the same timezone as the bugzilla server, and a less good but acceptable job if you aren’t.
  • Upgraded to exhibit 2.1.0 and now the numeric sliders with histograms work for me.  Woo!

Other notes:

The hg repo is here, as always.

DevMoXhibit: Exhibit on DevMo (Deki Wiki) results

Posted on by

devmo-search-customize-toolbar

The above screenshot is of a normal search query on DevMo for “customize toolbar”.  I see 2.5 results, and I honestly have no interest in the first item at all.  (It’s a page that only advanced DevMo authors would care about, for those who refuse to squint or click on images to see bigger versions of images.)

devmoxhibit-search-customize-toolbar-corrected

The above screenshot is of the same query using DevMoXhibit.  You will note you can see more things, and the first result from the other page is completely elided because we filter by default so that only “Real” result pages are shown.  (In general, I am not looking for talk pages or user pages or meta-pages.)

But enough about my interpretations of pictures, why don’t you:

Neat things we do that may not be immediately obvious:

  • We flatten the score into deciles, and then within each decile range we sort based on the view count for the page.  The theory is that, given equally likely results, the one that more people have looked at is probably more interesting to you, roughly speaking.
  • We use a simple heuristic to figure out the page type, as mentioned above (“Real”, “Talk”, “User”, etc.)
  • We try and hide all things related to the language, as we explicitly query on a language which means it’s just noise.  Right now, that language is always english, but the code uses a variable if you want to write the code to hook that up and expose it in the UI.
  • We produce a “smart” snippet.  The snippets provided by the search results naively will include “chrome” that is part of the document, which makes for a nearly useless snippet.  For example, take a gander at XUL/toolbar:
    • Plain old snippet:
      • « XUL Reference home    [ Examples | Attributes | Properties | Methods | Related ] A container which typically contains a row of buttons. It is a type of box that defaults to horizontal orientation. …
    • Smart snippet:
      • A container which typically contains a row of buttons. It is a type of box that defaults to horizontal orientation. …
  • We produce a sometimes over-zealous smart snippet.  If you were to keep reading both of those snippets, you would notice that the smart snippet eats a bit that the non-smart-snippet does not.  That is because the smart snippet is based on looking at a version of the snippet which has HTML tags in it, and then it tries to nuke those HTML tags out of existence using simple regexps.

Implementation notes:

  • This probably should work on other deki wikis if so adapted, but I don’t use any others, so YMMV.
  • We actually issue two search queries because there are two result formats that can be produced.  “xml” is an inexplicable mixture of too much data and too little data.  Namely, it does not tell you the tags on a document, which is basically the most useful piece of info, but it does tell you every link to and from that page (which we expose, although I doubt it will be useful enough to justify it).  It does give you a link to be able to get the tags, but that’s a costly operation when you have to perform it for each search result.  In contrast, “search” gives you the tags; they are only space-delimited, but that’s fine.  (“Inexplicable” may be a bit harsh; looking at the source, it’s just dumping the page info without further processing/lookups, but arguably it would be very useful if they made the effort to fetch that data.)
  • Because of cross-site XHR issues, this is not quite as hackable as I would like.  My demo server above is using mod_proxy (with a very specific constraint) to proxy the search to DevMo.  When I develop locally, I have to do the same thing.  Presumably if you are using Firefox 3.5 and devmo is set up correctly, then this would not be a problem.  But, 1) for no good reason, I only use Firefox 3.0 and 2) have no clue whether devmo is emitting the headers that would enable that to work.  I strongly encourage someone to look into #2 and fix it if not.
  • As with BugXhibit, the sliders are totally broken for me and it’s sad, but I left them in there in the hopes that they work for someone, somewhere.  Alternately, I would not complain if someone, somewhere, fixed them.

The hg repo is here.

BugXhibit: Exhibit on Bugzilla results

Posted on by

bugxhibit-timeline

I know it has sorta been done before (found via Bugzilla Fixup Wiki Page on a comment by faaborg), and I feel like there has to be another live version somewhere, but here we are.  BugXhibit is an MIT SIMILE Exhibit widget fronting a bugzilla.mozilla.org quicksearch query.

Click here to go to a BugXhibit page where you can enter your own query.  Enter “gloda” if you want to see what the screenshots are based on.  I feel like it would be improper of me to provide a link with a live query though.

Go visit the hg repo.  Or just download the source from the previous link.  Please improve!  (See the SIMILE Exhibit docs for how to do that.  It’s all really easy.)

bugxhibit-tile-view

Notes:

  • This uses bugzilla’s ctype=js for buglist.cgi.  It apparently has been around since 2003 (bug)!  And thanks to Gerv!  Perhaps not too surprisingly, the format of the results is not inert JSON but live JS code that builds a would-be-Array where each bug’s info is stored in an array.  What each element in the array stands for cannot be known from the results.  I find that using ctype=csv is a good way to get the headers.  Rather than doing that every time (cost concerns on the redundant query), I did it once for columnlist=all (which we always use) and stashed it in bugxhibit.js.  This is dangerous because it is brittle; if you try and use bugxhibit against a saved search someone made public, I at least got many fewer columns (despite columnlist=all), and things just don’t match.  Not to mention there is a “cf_blocking_fennec” flag in there that I feel like should not be there.
  • It looks pretty easy to have bugzilla produce more sane JSON output via a template (although the security code that logs you out for a js request still should run, so don’t forget buglist.cgi.)
  • Even with all columns exposed when using buglist.cgi, there are lots of interesting things that are not exposed.  For example, flags are not exposed via buglist.cgi, so faceting on whether things are blockers or wanted can’t be done.  Once you know the bug numbers from the query, you can obviously go fetch additional information, though I think that currently still needs to be XML format, but that’s not that hard.
  • The code is friendly and splits up the whiteboard and keyword things so it does what you would expect and is not stupid.
  • I made sliders for patch count and votes.  They don’t work for me anymore, and I see XUL wrapper anger (on Firefox 3.0.x), so, uh, don’t be surprised if they fall down.
  • The UI obviously sucks.  But it’s a proof of concept, and you are the internet!  You can do anything!

Better error reporting for the mozilla platform

Posted on by

old-and-busted-error-reporting

If you develop for the mozilla platform, you might be used to error messages like the above.  (Or you might wish you got error messages like the above…)  An uncaught javascript exception has resulted in a message in the error console as well as some equivalent stdout spew because it’s a debug build.  While any error is better than no error, it doesn’t exactly narrow down how we got there.

Wouldn’t it be nice if we got back-traces for these errors?

new-error-reporting-hotness

The future is now, people!  And it comes in the convenient form of a patch against the 1.9.1 branch, just like you always dreamed!  Also, an extension.

Currently (pre-patch), there are basically 3 ways scripting errors can show up in the platform:

  1. nsIScriptError instances.  These are what show up on the error console.  These have information equivalent to a single stack frame.
  2. nsIException instances.  These can provide a stack in the form of an nsIStackFrame chain (the same thing Components.stack gives you).  These get converted into nsIScriptError instances when it comes time to report them to the error console.  From a stack perspective, only XPConnect produces nsIException instances with stack traces, although you can make your own via Components.Exception.  A fundamental limitation of these stack traces is that they are only constructed from live JS call stacks, so if a JS exception has unwrapped its way to the top-level you are out of luck.
  3. JavaScript Error instances.  These have a private super-rich (it even knows arguments!) call-stack that can only be exposed as a string via the non-standard stack attribute.  XPConnect understands JS error reports (the ‘flat’ mechanism by which SpiderMonkey reports errors/exceptions to C++ code), but it has no clue about exceptions and their Error form of existence.  The exceptions in their error report guise are converted into nsIScriptError instances.

What the patch (on bug 493414) does is:

  • Introduce an nsIScriptErrorEx interface that extends nsIScriptError to provide a ‘location’ attribute like nsIException which is an nsIStackFrame.
  • Modify nsScriptError to implement the extended nsIScriptErrorEx.  Alternatively, I could have made XPConnect’s nsXPCException class implement nsIScriptError or nsScriptError also implement nsIException or something like that and not introduced nsIScriptErrorEx at all.
  • Modify all nsIScriptError-creation sites that I care about (I’m not looking at you, DOM workers) to try and provide or propagate existing nsIStackFrame information.
  • If a JS stack frame is not available, but an exception is in the form of a JS error, suck the call stack out of it.  Theoretically, this should not be a fallback but rather the default case, but it depends on some JS/XPConnect implementation details I am trying to avoid finding out about for now.
  • Modify the JS API to provide call stack sucking functionality.
  • Does various sketchy things to expose XPCJSStack::CreateStack from XPConnect to the error reporters in other modules.  If you thought the choice of creating nsIScriptErrorEx was sketchy, welcome to the Downtown East Side of dubious patches.  I expect there is no chance of it working on Windows because of this, and you may be out of luck on OS X.  Behold your comeuppance, popular platforms!

What the extension (repo) does is:

  • Add an nsIConsoleListener at app-startup that is aware of nsIScriptErrorEx and knows how to generate totally wicked 256-color ANSI escape sequences.
  • Not expose the stack traces in the error console.  The error console is for suckers who don’t have impossibly fast reflexes and a love of XON/XOFF flow control.
  • Only target Thunderbird.  Behold your comeuppance, all other mozilla applications!  (The extension wizard didn’t know how to do the thing that makes it work on all xulrunner-based things…  feel free to push a fixed install.rdf to my repo.)

I have logged bug 493414 to hold the patch and hopefully track the effort moving forward.

thunderbird, gloda, exptoolbar, protovis, paninaro, oh oh oh

Posted on by

exptoolbar-protovis-gloda-256

Thunderbird.  With the global database, gloda.  Using the exptoolbar extension.  Using the protovis javascript visualization library.  For reals!  Not a prank!  Just grab the most recent XPI or grab the repo.  And be using a nightly (beta 2 might work?)

What you are looking at:

  • The exptoolbar search results page, augmented with a visualization.
  • Each conversation with search results gets its own wedge.
    • Wedges can be distinguished because of the alternating background colors.
    • Conversations that you sent a message to will have a red shading to them.  The examples may be somewhat misleading because the account where a lot of my sent mail ends up is not part of the profile used to create the screenshots.
  • Each message is placed in its conversation wedge…
    • The radius is based on the ‘age’ of the message using a log-ish scale.  Interpolation is actually linear at each level (one day, one week, one month, three months, one year, 5 years, ‘forever’.)
    • The angular placement within the wedge is based on the author of the message.  Across all wedges the placement is the same.  This helps ‘bursty’ parts of conversations (which are extremely likely) be made more obvious, while also helping to provide some understanding of conversation dynamics.
  • Message shapes are determined by whether the message is starred (diamond), sent by a ‘popular’ contact (circle), or an unpopular one (cross).  The use of popularity is a temporary measure because current gloda in trunk does not cache address-book lookups, and they are expensive.  Once the new gloda search code lands with those changes, we can rely on the existence of an address book entry.  (Starring a contact using the new message reader adds them to your address book.)
  • Message opacity is determined by whether the message is a ‘hit’ or not.  All messages in a conversation are eventually retrieved, though initially we only have the hits.
  • Message color is determined by applied tags (using the closest tango color for the first tag), or whether the message is starred (closest tango color to yellow, where I think I had removed the yellow tango colors for some unknown reason, so we get green I guess).  It’s grey if the message has no tag or star.
  • The subject of the conversation is displayed in the wedge.

exptoolbar-protovis-seek-thunderbird-256

Things that are happy:

Things that are sad (aka caveats):

  • It would probably be better if the visualization was not radar-inspired.  Besides the perceptual reasons, the subjects are harder to read than they would be in an equivalent linear-styled visualization.
  • The visualization is not interactive.  protovis officially has no interaction support yet, but if you look in the (only available minified?) source, it’s almost there.  It might be entirely there, but it didn’t work for me immediately after a quick reading of the (indented) source.
  • There is some low probability failure that occurs during the visualization updating as gloda backfills the message collections.  If it happens on the last update, you can end up with a half-built visualization.  Re-running the search will generally resolve the issue.
  • The visualization does a pretty solid job of taking up all the screen real estate and has no way to be disabled, so you have to scroll past it every time.

Future work:

  • Interactivity.
  • Perhaps showing the gravatars for the people involved in a conversation at the outer rim of the wedge, positioning them based on the author positioning we determined.
  • Perhaps lose the radar motif.
  • Your thoughts / patches!