Category Archives: Thunderbird

teaser: Rich contextual information for Thunderbird mozmill failures

Posted on by

Sometimes Thunderbird mozmill unit tests fail.  When they do, it’s frequently a mystery.  My logsploder work helped reduce the mystery for my local mozmill test runs, but did nothing for tinderbox runs or developers without tool fever.  Thanks to recent renewed activity on the Thunderbird front-end, this has become more of a problem and so it was time for another round of amortized tool building towards a platonic ideal.

The exciting pipeline that leads to screenshots like you see in this post:

  • Thunderbird Mozmill tests run with the testing framework set to log semi-rich object representations to in-memory per-test buckets.
  • In the event of a test failure, the in-Thunderbird test framework:
    • Gathers information about the state of the system now that we know there is a failure and emit it.  This takes the form of canvas-based screenshots (using chrome privileges) of all open windows in the application, plus the currently focused element in the window.
    • Emits (up to) the last 10 seconds of log events from the previous test.
    • Emits all of the log events from the current test.
  • The python test driver receives the emitted data and dumps it to stdout in a series of JSON blobs that are surrounded by magical annotations.
  • A node.js daemon doing the database-based tinderboxpushlog thing (like my previous Jetpack/CouchDB work that found CouchDB to not be a great thing to directly expose to users and died, but now with node and hbase) processes the tinderbox logs for the delicious JSON blobs.
    • It also processes xpcshell runs and creates an MD5 hash that attempts to characterize the specific fingerprint of the run.  Namely, xpcshell emits lines prefixed with “TEST-” that have a regular form to describe when the pending test count changes or when specific comparison operations or failures occur.  It is assumed that tests will not comparison check values that are effectively random values, that the comparisons will tend to be deterministic (even if there are random orderings in the test) or notable when not deterministic, and thus that the trace derived from this filtering and hashing will be sufficiently consistent that similar failures should result in identical hashes.
    • Nothing is done with mochitests because: Thunderbird does not have them, they don’t appear to emit any context about their failures, and as far as I can tell, frequently the source of a mochitest failure is the fault of an earlier test that claimed it had completed but still had some kind of ripples happening that broke a later test.
  • A wmsy-based UI does the UI thing.

    The particular failure shown here is an interesting one where the exception is telling us that a popup we expected to open never opened.  But when we look at the events from the log, we can see that what happened is the popup opened and then immediately closed itself.  Given that this happened (locally) on linux, this made me suspect that the window manager didn’t let the popup acquire focus and perform a capture.  It turns out that I forgot to install the xfwm4 window manager on my new machine which my xvnc session’s xstartup script was trying to run in order to get a window manager that plays nicely with mozmill and our focus needs.  (Many window managers have configurable focus protection that converts a window trying to grab focus into an attention-requested notification.)

    This is a teaser because it will probably be a few more days before the required patch lands on comm-central, I use RequireJS‘ fancy new optimizer to make the client load process more efficient, and I am happy that the server daemons can keep going for long stretches of time without messing up.  The server and client code is available on github, and the comm-central patch from hg.

    Thunderbuddy: Weave-synchronized Thunderbird contacts on Android

    Posted on by

    I got my first modern Android phone at the end of last week and I figured it would be a good idea to get over the activation energy hump so it’s easy to do incremental mozilla hacking on it.  Thanks to a lot of work by other people, this turned out to be pretty easy.  It would have been even easier if the Android emulator ran at anything remotely close to real time or if my Samsung Galaxy S Vibrant (Bell Canada model) running 2.1 did not have a habit of locking up all the time.  (And this is before my code touched it, mind you.  I’m half-hoping my device is a lemon.  If anyone knows how to tell a lemon from lockups-as-usual, please let me know.)

    The screenshot shows my Android device running the app.  It’s a super trivial wmsy UI showing contacts from my address book pulled down from a weave server that were put there by Thunderbird.

    Anywho, props to:

    • Phonegap, the HTML/JS way to write mobile apps!  Major awesome.
    • weaveclient-chromium, a slightly bitrotted pure JS weave client by Philipp von Weitershausen / philikon (now of the mozilla weave/sync team) which built on the weaveweb project by Anant Narayanan and pure JS crypto work by the people mentioned in the README.
    • weaver and weave-ext by Shane Caraveo, which make weave happily run in Thunderbird and have it propagate the contents of the address book.
    • The Mozilla Weave/Firefox Sync team who made it easy and practical for software like Thunderbird to partake in the encrypted synchronization revolution.
    • RequireJS, CommonJS loader to the stars.

    The relevant repos for those interested, are:

    • weaveclient-js: This is a fork of weaveclient-chromium that ditches the chromium bits, makes things CommonJS/Asynchronous Module Definition happy, and slightly factors out the encryption so that thunderbuddy can provide ‘native’ accelerated encryption support on android.
    • thunderbuddy: The phonegap android app repo proper.  The only really notable thing at this point is the custom Java class that implements and exposes faster ‘native’ encryption methods.  (Thunderbuddy can also just be used as a webpage on any reasonably capable browser with good JS performance.)

    It’s worth noting that the goal is not actually to perform contact synchronization with Android.  There are already a ton of ways to synchronize your Thunderbird contacts with gmail and from there your phone.  The goal is to let other interesting data and meta-data propagate.  I just picked contacts for this example because Shane already had the data propagating.

    Documentation for complex things (you don’t basically already understand)

    Posted on by

    The Problem

    One of my focuses at MoMo is to improve the plight of Thunderbird extension developers.  An important aspect of this is improving the platform they are exposed to.  Any platform usually entails a fair amount of complexity.  The trick is that you only pay for the things that are new-to-you, learning-wise.

    The ‘web’ as a platform is not particularly simple; it’s got a lot of pieces, some of which are fantastically complex (ex: layout engines).  But those bits are frequently orthogonal, can be learned incrementally, have reams of available documentation, extensive tools that can aid in understanding, and, most importantly, are already reasonably well known to a comparatively large population.  The further you get from the web-become-platform, the more new things you need to learn and the more hand-holding you need if you’re not going to just read the source or trial-and-error your way through.  (Not that those are bad ways to roll; but not a lot of people make it all the way through those gauntlets.)

    I am working to make Thunderbird more extensible in more than a replace-some-function/widget-and-hope-no-other-extensions-had-similar-ideas sort of way.  I am also working to make Thunderbird and its extensions more scalable and performant without requiring a lot of engineering work on the part of every extension.  This entails new mini-platforms and non-trivial new things to learn.

    There is, of course, no point in building a spaceship if no one is going to fly it into space and fight space pirates.  Which is to say, the training program for astronauts with all its sword-fighting lessons is just as important as the spaceship, and just buying them each a copy of “sword-fighting for dummies who live in the future” won’t cut it.

    Translating this into modern-day pre-space-pirate terminology, it would be dumb to make a super fancy extension API if no one uses it.  And given that the platform is far enough from pure-web and universally familiar subject domains, a lot of hand-holding is in order.  Since there is no pre-existing development community familiar with the framework, they can’t practically be human hands either.

    The Requirements

    I assert the following things are therefore important for the documentation to be able to do:

    • Start with an explained, working example.
    • Let the student modify the example with as little work on their part as possible so that they can improve their mental model of how things actually work.
    • Link to other relevant documentation that explains what is going on, especially reference/API docs, without the user having to open a browser window and manually go search/cross-reference things for themselves.
    • Let the student convert the modified example into something they can then use as the basis for an extension.

    The In-Process Solution: Narscribblus

    So, I honestly was quite willing to settle for an existing solution that was anywhere close to what I needed.  Specifically, the ability to automatically deep-link source code to the most appropriate documentation for the bits on hand.  It has become quite common to have JS APIs that take an object where you can have a non-trivial number of keys with very specific semantics, and my new developer friendly(-ish) APIs are no exception.

    Unfortunately, most existing JavaScript documentation tools are based on the doxygen/JavaDoc model of markup that:

    • Was built for static languages where your types need to be defined.  You can then document each component of the type by hanging doc-blocks off them.  In contrast, in JS if you have a complex Object/dictionary argument that you want to hang stuff of of, your best bet may be to just create a dummy object/type for documentation purposes.  JSDoc and friends do support a somewhat enriched syntax  like “@param arg.attr”, but run into the fact that the syntax…
    • Is basically ad-hoc with limited extensibility.  I’m not talking about the ability to add additional doctags or declare specific regions of markup that should be passed through a plugin, which is pretty common.  In this case, I mean that it is very easy to hit a wall in the markup language that you can’t resolve without making an end-run around the existing markup language entirely.  As per the previous bullet point, if you want to nest rich type definitions, you can quickly run out of road.

    The net result is that it’s hard to even describe the data types you are working with, let alone have tools that are able to infer links into their nested structure.

    So what is my solution?

    • Steal as much as possible from Racket (formerly PLT Scheme)’s documentation tool, Scribble.  To get a quick understanding of the brilliance of Racket and Scribble, check out the quick introduction to racket.  For those of you who don’t click through, you are missing out on examples that automatically hyperlink to the documentation for invoked methods, plus pictures capturing the results of the output in the document.
      • We steal the syntax insofar as it is possible without implementing a scheme interpreter.  The syntax amounts to @command[s-expression stuff where whitespace does not matter]{text stuff which can have more @command stuff in it and whitespace generally does matter}.  The brilliance is that everything is executed and there are no heuristics you need to guess at and that fall down.
      • Our limitation is that while Racket is a prefix language and can use reader macros and have the entire documents be processed in the same fashion as source code and totally understood by the text editor, such purity is somewhat beyond us.  But we do what we can.
    • Use narcissus, Brendan Eich/mozilla’s JS meta-circular interpreter thing, to handle parsing JavaScript.  Although we don’t have reader macros, we play at having them.  If you’ve ever tried to parse JavaScript, you know it’s a nightmare that requires the lexer to be aware of the parsing state thanks to the regexp syntax.  So in order for us to be able to parse JavaScript inline without depending on weird escaping syntaxes, when parsing our documents we use narcissus to make sure that we parse JavaScript as JavaScript; we just break out when we hit our closing squiggly brace.  No getting tricked by regular expressions, comments, etc.
    • Use the abstract interpreter logic from Patrick Walton‘s jsctags (and actually stole its CommonJS-ified narcissus as the basis for our hacked up one too) as the basis for abstract interpretation to facilitate being able to linkify all our JavaScript code.  The full narcissus stack is basically:
      • The narcissus lexer has been modified to optionally generate a log of all tokens it generates for the lowest level of syntax highlighting.
      • The narcissus parser has been modified to, when generating a token log, link syntax tokens to their AST parse nodes.
      • The abstract interpreter logic has been modified to annotate parse nodes with semantic links so that we can traverse the tokens to be able to say “hey, this is attribute ‘foo’ in an object that is argument index 1 of an invocation of function ‘bar'” where we were able to resolve bar to a documented node somewhere.  (We also can infer some object/class organization as a result of the limited abstract interpretation.)
      • We do not use any of the fancy static analysis stuff that is going on as of late with the DoctorJS stuff.  Automated stuff is sweet and would be nice to hook in, but the goal here is friendly documentation.
      • The abstract interpreter has been given an implementation of CommonJS require that causes it to load other source documents and recursively process them (including annotating documentation blocks onto them.)
    • We use bespin as the text editor to let you interactively edit code and then see the changes.  Unfortunately, I did not hook bespin up to the syntaxy magic we get when we are not using bespin.  I punted because of CommonJS loader snafus.  I did, however, make the ‘apply changes’ button use narcissus to syntax check things (with surprisingly useful error messages in some cases).

    Extra brief nutshell facts:

    • It’s all CommonJS code.  The web enabled version which I link to above and below runs using a slightly modified version of Gozala‘s teleport loader.  It can also run privileged under Jetpack, but there are a few unimplemented sharp edges relating to Jetpack loader semantics.  (Teleport has been modified mainly to play more like jetpack, especially in cases where its aggressive regexes go looking for jetpack libs that aren’t needed on the web.)  For mindshare reasons, I will probably migrate off of teleport for web loading and may consider adding some degree of node.js support.  The interactive functionality currently reaches directly into the DOM, so some minor changes would be required for the server model, but that was all anticipated.  (And the non-interactive “manual” language already outputs plain HTML documents.)
    • The web version uses a loader to generate the page which gets displayed in an iframe inside the page.  The jetpack version generates a page and then does horrible things using Atul‘s custom-protocol mechanism to get the page displayed but defying normal browser navigation; it should either move to an encapsulated loader or implement a proper custom protocol.

    Anywho, there is still a lot of work that can and will be done (more ‘can’ than ‘will’), but I think I’ve got all the big rocks taken care of and things aren’t just blue sky dreams, so I figure I’d give a brief intro for those who are interested.

    Feel free to check out the live example interactive tutorialish thing linked to in some of the images, and its syntax highlighted source.  Keep in mind that lots of inefficient XHRs currently happen, so it could take a few seconds for things to happen.  The type hierarchy emission and styling still likely has a number of issues including potential failures to popup on clicks.  (Oh, and you need to click on the source of the popup to toggle it…)

    Here’s a bonus example to look at too, keeping in mind that the first few blocks using the elided js magic have not yet been wrapped in something that provides them with the semantic context to do magic linking.  And the narscribblus repo is here.

    Thunderbird Quick Filter Bar extensions, they’re a thing!

    Posted on by

    The previously discussed Quick Filter Bar interface landed in time for Thunderbird (Lanikai) 3.1 beta 2 (whose release is real-soon-now).  Although the Quick Filter Bar already contains dangerously high levels of awesome, we made sure to make it extensible so you can cram even more awesome in.

    As an example, I have created an extension that enables you to ‘pivot’ based on the currently selected message(s).

    In its most straightforward manner of operation, you can click on an e-mail address in the message reader header and pivot by that e-mail address.  Out of the box, this will show you all the messages in the current folder sent by that user.  You can also (or only) show messages where that user is on the to/cc lines by toggling the buttons on the expando bar appropriately.

    You can also constrain the display to only show messages within some time interval of the message(s) in question.

    The more confusing way to access the pivot functionality is to simply toggle the facet on the quick filter bar.  When you toggle the filter on, we walk all of the selected messages and build up a list of the observed e-mail addresses for all of the senders and all of the recipients.  One trick is that we filter out any e-mail addresses associated with your own account in order to avoid the filter becoming too inclusive.  We save those e-mail addresses into our state.  We also walk all the messages and find the time range that the messages cover and save that.  These are used as the basis for the specific constraints you can enable from the expando bar.  Because the values are saved, the filter does not change when you change your selected messages.  You need to toggle the pivot filter off and on again (or use the ‘pivot’ option on the email address popup) to change the data we are pivoting on.

    The extension can be found on AMO and in source form.  In a forward-looking move that is probably complicating things for now, I used the Jetpack SDK and its XUL extension support to implement the extension.  In practice, all the example uses Jetpack for is its CommonJS module system and the standard mozilla-style JS modules would have been sufficient.  All the UI stuff is done purely in XUL overlays bound to the backing code using addEventListener.

    The broad strokes for implementing a Quick Filter Bar extension are:

    1. Overlay your toolbarbutton into the quick filter bar, preferably the #quick-filter-bar-collapsible-buttons box.  If you have an expando bar you want to contribute, you probably want to put that into the #quick-filter-bar-expando.  As things currently stand, multiple expando bars being uncollapsed at the same time will probably turn out poorly, so you may need to be pragmatic and put your bar outside that box or do something clever.
    2. Import resource:///modules/quickFilterManager.js, implement your filter, and register it with QuickFilterManager.defineFilter() (whose documentation explains the interface you need to conform to).

    We will likely not be doing up full documentation for this extension point beyond the in-code documentation and in-tree/extension examples.  This is an advanced extension point by virtue of it touching nsIMsgDBHdrs, nsIMsgDBViews, nsIMsgSearchSessions, and friends, all of which have sharp edges.  The way forward with soft, human-friendly documented abstractions is gloda and that’s where (my) effort will be concentrated.  The awesome in this extension point is making it practical for people who already have the scars to utilize the full power of the message view/traditional search subsystem without losing a limb and do so in a previously impossibly short period of time.

    PS: I made the icon!

    work-in-progress tooling: feeding GWT SpeedTracer systemtapped mozilla performance data

    Posted on by

    Now that Thunderbird 3.1 is string/feature-frozen it’s time to focus on performance.  This post is just to make sure that people who are also working on similar things know what I’m up to.  For example, there’s some very exciting work going on involving adding a startup timeline to mozilla-central that everyone keeping up-to-date with mozilla performance work should be aware of.  In summary, no results yet, but soon!

    The big bullet points for now are:

    • GWT SpeedTracer is a very impressive tool.
    • I have seriously cleaned up / automated my previous SystemTap script work for mozilla.  Invoking python chewchewwoowoo.py speedtracer/mozspeedtrace.stp `pgrep thunderbird-bin` does the following:
      • generates a version of the source script with any line statements fixed up based on one or more line seeks in a file.
      • builds the command line and invokes the system tap script.
      • invokes a post-processing script specified in the tap file, feeding it address-translation helpers and systemtap bulk logging per-cpu fusion.
    • mozspeedtrace.stp and its post-processor produce an HTML file with embedded JSON that the SpeedTracer Chrome extension recognizes as one of its own.  (SpeedTracer runs in Firefox through the GWT development mode, but being very new to GWT I am not yet clear on if/how to be able to wrap it up to be used as a webapp from static data.)
    • My mapping of events as recorded by my SystemTap probes to SpeedTracer events is somewhat sketchy, but part of that is due to the limited set of events and their preconfigured display output.  (I believe it turns out I can generate optional data like fake stack traces to be able to expand the set of mapped events without having to modify SpeedTracer.)  This is way basically everything in the screenshot is a yellow ‘JavaScript Callback’.
    • I have not implemented probes for all of the event types defined by SpeedTracer, I have not ported all of my previous probes yet, and there are still some new probes yet to add.  Happily, most of the hard work is already in the can.
    • The motivation behind this is very different from the startup timeline.  Thunderbird uses mozilla-1.9.2, not mozilla-central, and my primary investigatory focus is memory usage in steady-state after start-up, not the time it takes to start up.  (We do, of course, want to improve performance where possible.)  It just so happens that in order to usefully explain who is allocating memory we also have to know who is active at the time / why they are active, and that provides us with the performance timeline data.
    • This work is not likely to be useful for per-tab/webpage performance data gathering… just whole-program performance investigation like Thunderbird needs.
    • This is a work-in-progress, but the focus is on improving Thunderbird’s performance issues, not building tooling.  As such, I may have to bail on SpeedTracer in favor of command-line heuristic analysis if I can’t readily modify SpeedTracer to suit my needs.  I picked SpeedTracer partially to (briefly) investigate GWT, but from my exploration of the code thus far, the activation energy required may be too great.

    performance annotated SQLite EXPLAINation visualizations using systemtap

    Posted on by

    For the Thunderbird 3.1 release cycle we are not just fixing UX problems but also resolving various performance issues.  Building on my previous work on a visualization of SQLite opcode control flow graphs using graphviz I give you… the same thing!  But updated to more recent versions of SQLite and integrating performance information retrieved through the use of systemtap with utrace.

    In this case we are using systemtap to extract the number of times each opcode is executed and the number of btree pages that are requested during the course of executing the opcode.  (We do not differentiate between cache hits and misses because we are looking at big-O right now.)  Because virtual tables (like those used by FTS3) result in nested SQLite queries and we do not care about analyzing the queries used by FTS3, we ignore nested calls and attribute all btree page accesses to the top-level opcode under execution.

    Because the utility of a tool is just as dependent on ease of use as its feature set, I’ve cleaned things up and made it much easier to get information out of Thunderbird/gloda with this patch which should land soon and provides the following:

    • The gloda datastore will write a JSON file with the EXPLAINed results of all its SQL queries to the path found in the preference mailnews.database.global.datastore.explainToPath.  This preference is observed so that setting it during runtime will cause it to create the file and begin explaining all subequently created queries.  Clearing/changing the preference closes out the current file and potentially opens a new one.
    • Gloda unit tests will automatically set the preference to the value of the environment variable GLODA_DATASTORE_EXPLAIN_TO_PATH if set.
    • A schema dump is no longer required for meta-data because we just assume that you are using a SQLite DEBUG build that tells us everything we want to know about in the ‘comment’ column.
    • grokexplain.py now uses optparse and has more internal documentation and such.

    So what do the pretty pictures show?

    • Before: A gloda fulltext query search retrieves all of the results data before applying the LIMIT.  This results in a lot more data transplanted into temporary results tables than we will end up using; wasted bookkeeping.  Additionally, we incur the row lookup costs for both our messages data storage table and our fulltext messagesText table for all hits, even the ones we will not return.  (Noting that there was no escaping hitting both tables since we were using offsets() and it hits the fulltext table as part of its operation.)
    • After: We perform an initial query phase where we minimize wasted bookkeeping by only retrieving and using the bare minimum required to compute the LIMITed list of document id’s.  Additionally, by use of the FTS3 matchinfo() function instead of the offsets() function we are able to avoid performing row lookup on the messagesText table for results that will not be returned to the user.  Use of the matchinfo() function requires a custom ranking function which allows us to be somewhat more clever about boosting results based on fulltext matches too.
    • The poor man’s bar graphs in the pictures are expressing a hand-rolled logarithmic scale for the number of invocations (left bar) and number of btree pages accessed (right bar).  On the right side of each line the actual numbers are also presented in the same order.  The goal is not to convey good/bad so much as to draw the eye to hot spots.

    Notes for people who want more to read:

    • SQLite has built-in infrastructure to track the number of times an opcode is executed as well as the wall-clock time used; you do not have to use systemtap.  It’s a conditional compilation kind of thing, just -DVDBE_PROFILE and every statement you execute gets its performance data appended to vdbe_profile.out when you reset the statement.  It doesn’t do the btree pages consulted trick, but it’s obviously within its power with some code changes.
    • Our use of systemtap is primarily a question of discretionary reporting control, the ability to integrate the analysis across other abstraction layers, and various build concerns.  The JSON output is not a driving concern, just something nice we are able to do for ourselves since we are formatting the output.
    • The tool has really crossed over into the super-win-payoff category with this fix investigation.  (Having said that, I probably could have skipped some of the data-flow stuff last time around.  But these projects are both for personal interest and amusement as well as practical benefit, so what are you gonna do?  Also, that effort could pay off a bit more by propagating comments along register uses so that the LIMIT counter register r8 and the compute-once r7 in the after diagram would not require human thought.)

    References:

    • The grokexplain repo.  Used like so: python grokexplain.py –vdbe-stats=/tmp/glodaNewSearchPerf.json /tmp/glodaNewSearchExplained.json -o /tmp/glodasearch-newcheck
    • The systemtap script in its repo.  Used like so: sudo stap -DMAXSTRINGLEN=1024 sqlite-perf.stp /path/to/thunderbird-objdir/mozilla/dist/lib/libsqlite3.so > /tmp/glodaNewSearchPerf.json
    • The bug with the gloda explain logic patch and the improved SQL query logic.  I also used the tooling to fix another (now-fixed) bug, but that problem turned out to be less interesting.

    more systemtap mozilla event loop awareness

    Posted on by 
    ====================                                             ms    #
----- Event Loop:
  nsInputStreamReadyEvent                                      7119   11
----- Input Events:
  nsInputStreamPump                                            7119   11
----- Pump Events:
  thunk:nsNNTPProtocol::QueryInterface(...)                    7119   11

====================                                             ms    #
----- Event Loop:
  nsInputStreamReadyEvent                                      6284   15
  nsTimerEvent                                                  242  123
----- Timers:
  nsUITimerCallback                                             121    2
----- Input Events:
  nsInputStreamPump                                            6284   15
----- Pump Events:
  thunk:nsNNTPProtocol::QueryInterface(...)                    6283   15

    As of late, I’ve noticed that Thunderbird has been pegging one of my CPU cores periodically in ways not correlated with gloda activity, the usual scapegoat.  With some enhancements to my systemtap script from last time, I can now see that it apparently takes something like 13 seconds of dominating the event loop to perform the check for new messages in my 26 subscribed news folders on news.mozilla.org.  Yowch.  At least this is on a debug build!

    Changes are that we now cover input stream ready events, input pump listeners, and proxied objects (which lets us see what Thunderbird IMAP gets up to).  Also, the output is more concise and only shows things that take at least 100ms of aggregate wall clock time.

    Script invocation currently wants to look like this:
    sudo stap -v moz-event-loop.stp /path/to/thunderbird/objdir/mozilla/dist/lib/libxpcom_core.so /path/to/thunderbird/objdir/mozilla/dist/bin/components/libnecko.so | ../addrsymfilt.py `pgrep thunderbird-bin`

    Thunderbird ĝLȬdÅ full text search tokenizer now supports accent folding, non-ASCII case-folding, and more!

    Posted on by

    Thanks to the efforts of Makoto Kato (jp-blog en-blog twitter) whom you may remember as the bringer of CJK bi-gram tokenization, I have just pushed tokenizer support for case-folding, accent-folding, and voiced sound mark magic to comm-central.  The net result is that searches for “ĝLȬdÅ” will now find “gloda” and vice versa.  Less crazy searches should also be improved.

    Starting tomorrow, Thunderbird (in the guise of Lanikai) 3.1 nightlies will have this capability and it will also show up in the next beta (beta 2).  No action on your part is required; when you start running a build with this capability your existing gloda database will get blown away and indexing will start from scratch.  If you go back to an older version for some reason after having used the updated build, the old build will also decide to blow away your database, so try to avoid making a habit of switching between them.

    We also have some significant performance enhancements related to gloda search and indexing slated for beta 2, not to mention* usability enhancements to avoid the dreaded gloda search/quick search toggle dance.

    * This expression is weird given that I go on to mention the allegedly unmentioned, but it still makes more sense than “all but X”.

    Thunderbird Message Filter Bar Prototype Extension, check it.

    Posted on by

    What was just the quick-search box in Thunderbird 2 is now also the home to global search in Thunderbird 3.  This hasn’t turned out splendidly, although we didn’t expect that it would.  Some people think quick-search is gone because they do not realize you can change the modes of the search box.  Other people are hardcore and know how to switch between the modes, they just don’t like all the clicking.

    So we’re thinking about splitting the quick-search out into its own separate box.  In addition, we’re trying to expose a lot of the power of the “mail views” system.  You may know “mail views” as that boxy thing that lived with the quick-search box above the thread pane in Thunderbird 1.5 but then became something you had to customize onto the toolbar at some point.  It looks like this:

    Thunderbird has a very nice search subsystem under the hood that powers quick-search, “advanced search”, virtual folders, etc.  Mail views was and is the mechanism that allowed you to define arbitrary searches and use them as filters on any folder.  Unfortunately it’s not a smooth operator and its defaults have some issues.  There’s no “starred” filter unless you define your own, there’s no “any tag” filter unless you define your own, and “People I Know” only checks one of the two address books you are likely to have in your profile.  Even with those defined, you’re looking at 3 clicks to get to most things.

    So the message filter bar is also trying to bring one-click access to these things you might care about.  In the top screenshot, that’s what you’re looking at.  Starred messages, messages from people in any of your local address books, messages with tags, and messages with attachments are all at your fingertips.  The new quick-search location over on the right works right with them.  And if you love mail views and all the clicking-finger muscle strength it helps build, it works with mail views too!

    One bit of polish that I’m hoping people like and performs sufficiently well is the tags case.  When you haven’t clicked on the tags icon, the bar in the screenshot does not show that bit with the tags.  When you click on it, it 1) filters the visible messages to messages with tags, and 2) figures out what tags are on those messages and populates that bar with those tags.  You can then click on any of the tags to stop including messages with that tag (and none of the other tags still selected).

    In any event, if you are interested, the prototype is being developed as an extension that you can find here on AMO (sandboxed) if you like pre-built XPIs and here in hg if you like source code.  It is very prototype-y at the current moment.  Keybindings aren’t there, localization/accessibility is not there, being able to make the filter bar go away isn’t there, etc.  We will iterate on things and productize assuming the concept works out.  Just be aware that I don’t believe sandboxed plugins auto-update, so you if you’re really interested you might need to keep an eye on the AMO page or the repo.

    UPDATE: I have nuked the add-on from AMO since Thunderbird 3.1 beta 2 is now available.