Caio Romão

Vim Plugin: Query Command Complete

2012-07-15T22:00:00+02:00

I’ve just released a new Vim plugin to improve how I’ve been writing my e-mails. Every once in a while I remember I need to include someone in that e-mail I’ve started writing a few minutes before and that’s where using a normal text editor to write falls short: by default, there’s no integration between it and any sort of address book.

Note that I’m not talking about the usual workflow imposed by the command line clients, where you select the recipients and then you write your message. In this case, just setting up a query command (or analogous on your MUA of choice. I’m pretty sure it’s a fairly standard feature).

Well, this plugin’s intention is setting up a completion system that queries an external command using the same kind of tools that are used for managing your address book. As long as the output of such command is easily parseable, you can use it with this plugin. Here is how it works:

You setup the g:qcc_query_command option to the call to your address book (say, abook --mutt-query) and enable the completion on the filetype of your choice:

# Standard setup for mutt
let g:qcc_query_command = 'abook --mutt-query'
# You might consider using `omnifunc` if you have a completion
# system plugin like neocomplcache
au BufRead /tmp/mutt* setlocal completefunc=QueryCommandComplete

Now whenever you realise you have to fiddle with the headers that take e-mail addresses as parameters (From, To, Cc, etc), you can do it without leaving your Vim session. For example if you have the following in your line (| being the cursor position):

From: Foobar <foo@bar.com>, cai|

And then you trigger user completion (usually C-x C-u), Query Command Complete will go ahead and call abook --mutt-query cai and present the results back to you in that glorious wildmenu we all know and love.

There’s some configuration goodness also, so one can even adapt it to work with the most bizarre things:

g:qcc_line_separator: Separator for each entry in the result from the query. Default: '\n'.
g:qcc_field_separator: Separator for the fields of an entry from the result. Default: '\t'.
g:qcc_pattern: Pattern used to match against the current line to decide whether to call the query command. Default: '^\(To\|Cc\|Bcc\|From\|Reply-To\):'.

So, enough marketing. Give Query Command Complete a go and if you find any issues or want to contribute, check the project at Github. Have fun!

Exercise: CLI Fuzzy Clock

2012-03-28T00:00:00+02:00

While backing up stuff during preparations for my move abroad (more on that later), I found a couple of tiny projects using Autotools and realized I remembered almost nothing about this build system and then I decided I should do a quick refresher project with it.

Since remembering stuff is not as fun as learning, I jumped on the testing-in-c bandwagon and decided I’d learn and use the Check unit testing framework.

So, why a fuzzy clock? No idea. It was the first really simple but not completely useless project that came to mind.

You can check the source code and also download the initial release of this project.

How "puts" works in Ruby

2011-10-21T00:00:00+02:00

Most object oriented programming languages provide an interface that an object can implement to describe itself as a string, so when it is given as an argument to a “printing” function, it gets converted/coerced to it’s desired format.

In Ruby, this is achieved by implementing the to_s method, so if you define a class implementing this method, puts will behave like this:

>> class MyClass
>>   def to_s
>>     "<MyClass Instance>"
>>   end
>> end
=> nil
>> instance = MyClass.new
=> <MyClass Instance>
>> puts instance
<MyClass Instance>
=> nil

So everything is well and works as expected until you suddenly decide to subclass the Array or the String class:

>> class MyString < String
>>   def to_s
>>     "<MyString #{super}>"
>>   end
>> end
=> nil
>>
>> class MyArray < Array
>>   def to_s
>>     "<MyArray instance>"
>>   end
>> end
=> nil
>>
>> mystring = MyString.new("Hello World!")
=> "Hello World!"
>>
>> myarray = MyArray.new
=> []
>>
>> puts mystring
Hello World!
=> nil
>> puts myarray
=> nil

Wait, what!? As you can see, I’ve implemented the to_s function but it didn’t get called when I passed the instances to the puts function. That means I did something wrong, right?

>> mystring.to_s
=> "<MyString Hello World!>"
>> myarray.to_s
=> "<MyArray instance>"

Well, apparently not. So what kind of trickery is going on here? I’ve implemented the necessary method but it’s not being called for those specific subclasses, so it’s bound to be an implementation detail of the function we are calling. Let’s take a look at how puts is implemented in Ruby ¹.

Opening up the io.c file and searching for “puts” we end up finding a declaration of the rb_io_puts function:

VALUE
rb_io_puts(int argc, VALUE *argv, VALUE out)
{
    int i;
    VALUE line;

    /* if no argument given, print newline. */
    if (argc == 0) {
        rb_io_write(out, rb_default_rs);
        return Qnil;
    }
    for (i=0; i<argc; i++) {
        if (TYPE(argv[i]) == T_STRING) {
            line = argv[i];
            goto string;
        }
        line = rb_check_array_type(argv[i]);
        if (!NIL_P(line)) {
            rb_exec_recursive(io_puts_ary, line, out);
            continue;
        }
        line = rb_obj_as_string(argv[i]);
    string:
        rb_io_write(out, line);
        if (RSTRING_LEN(line) == 0 ||
            !str_end_with_asciichar(line, '\n')) {
            rb_io_write(out, rb_default_rs);
        }
    }

    return Qnil;
}

The comment block above the function definition already explains what happens, but let’s understand the function and why it behaves the way it does when we call it with our custom Array and String subclasses.

TYPE is a macro that expands to rb_type, which ultimately checks the .flags field from the object cast as struct RBasic to identify it’s type. So, for the string behavior it’s simple:

TYPE(argv[i]) == T_STRING is True, which makes the code jump to the string label and simply write it’s contents.

For the array case, the code calls rb_check_array_type (from array.c), which cascades to testing TYPE(argv[i]) == T_ARRAY && T_ARRAY != T_DATA. That results in true, so the code enters the if (!NIL_P(line)) block, calling io_puts_ary (that ends up calling rb_io_puts for each item of the array) and goes to handle the next parameter.

Now, only if both tests above fails the interpreter ends up calling rb_obj_as_string, which, as you may have guessed, is responsible for calling the higher level to_s implementation.

So now you might be thinking “if I don’t subclass Array nor String ruby will certainly call my to_s implementation, right?”. Well, wrong! :)

If you’ve been following along what I’ve written with Ruby’s source code, you might have noticed the following:

VALUE
rb_check_array_type(VALUE ary)
{
    return rb_check_convert_type(ary, T_ARRAY, "Array", "to_ary");
}

Notice the last parameter being given (to_ary)? It’s a well known friend of rubyists: it’s a method that converts an object to array.

Without digging deeper in the code, what might this mean? Let’s do a little test. Remember our first example, the MyClass implementation? We’ll build up on that:

>> class MyClass
>>   def to_s
>>     "<MyClass Instance>"
>>   end
>> end
=> nil
>> instance = MyClass.new
=> <MyClass Instance>
>> puts instance
<MyClass Instance>
=> nil
>> class MyClass
>>   def to_ary
>>     %W(MyClass Instance as Array)
>>   end
>> end
=> nil
>> puts instance
MyClass
Instance
as
Array
=> nil

Weird isn’t it? Turns out that rb_check_array_type returns a valid pointer if the parameter implements the to_ary method. This means that whenever an object has this method implemented, it will be treated as an array by puts and the to_s method will never be called by it.

And that’s it, phew! To summarise: puts does call to_s as long as you don’t:

Subclass from String nor Array
Implement the to_ary method

Note: This post idea comes from a question I’ve answered on a Portuguese stackoverflow-like website some months ago.

I’m using the C reference implementation for Ruby, version 1.9.2-p290, obtainable at http://ruby-lang.org/.
↩

Musings About RPM Upgrades

2011-09-14T00:00:00+02:00

TL;DR Version: We’ve screwed up for not really understanding how the packaging system we used worked internally.

I currently work on a project where we have to go full stack: from planning, coding and testing to packaging, shipping and doing some legal paper work. One of the areas that gives the most headaches (besides the legal hassle, obviously) is the packaging (we’re only dealing with RPM for now). Here I’ll describe an RPM default behavior which has bitten us recently and how we’ve used it for our own good.

RPM packages support scripting at some points during installation: before and after installation and removal - namely:

%pre: kicks in before installing the package - this stage is commonly used to prepare the environment before the install, such as creating directories and setting permissions.
%post: runs after the package has been installed, typically used to run configuration scripts, like creating the administrator credentials for a database or setting up the service to start at boot.
%preun: happens before a package removal (or upgrade), used to stop dependant services and kill the application being uninstalled.
%postun: the last step of a removal (or, again, an upgrade), sometimes used for some leftover cleanup like purging empty directories.

I won’t go too deep on the workings of RPM as this is not the scope of this post, but if you’re interested check out Maximum RPM: it’s the most complete RPM guide you’ll ever get. As a bare minimum, take a quick look at here and here.

So, what would happen behind the scenes during a package upgrade? It’s natural (at least for me and my team) to think that the old version package will be removed and then the new version would be installed -meaning that the sections described previously would be called in the following order: %preun, %postun, %pre, %post. Well, turns out this is not what happens and this is why we were bitten. Twice.

Bite #1: Our Services Were All Stopped After An Upgrade

Following the “natural” chain of thought, we’d included in our package’s %preun section instructions to stop the services we setup so that it wouldn’t be running during the upgrade and risk breaking everything.

This definitely didn’t break anything and since we provide a front-end for upgrading our packages which triggers a restart after everything, we didn’t spot the issue soon. Only when testing a command line upgrade it was noticed that our services were completely stopped.

The fix for it was simple: we removed the commands to stop our services in the %preun section during an upgrade and everything went better than expected.

Bite #2: Files Generated Dynamically Were Being Removed

Due to some internal reasons one of the packages we ship switched from including some files statically in the RPM package (defined in the %files section) to automatically generating them during the install process.

With this change, after an upgrade from our old and more declarative package to this new dynamic one the files that were being generated ended up being removed after the upgrade (we had confirmed they were definitely being created during the setup).

The rather-ugly-but-simple-fix was to create an intermediary package with static files with names different from the ones being auto generated.

How An Upgrade REALLY Works

So, how come those fixes worked and - more importantly - why the heck were they happening in the first place?

Quoting from this developerWorks article, what happens during a package upgrade is the following:

Run the %pre section of the RPM being installed.

Install the files that the RPM provides.

Run the %post section of the RPM.

Run the %preun of the old package.

Delete any old files not overwritten by the newer version. (This step deletes files that the new package does not require.)

Run the %postun hook of the old package.

What this means is that the old package removal happens after the installation of the new one. If you think about it, it kind of makes sense since we don’t want to touch the configuration files the user has changed (or rather, we don’t want to force the user to reconfigure her service after simple updates). If it were up to me I wouldn’t do it like this, but anyway.

Let’s understand each issue and why the fix worked:

The issue with services being stopped (Bite #1) is fairly simple to understand once you read about the upgrade order above: you must not stop services on the %preun and %postun sections during an update/upgrade.
Issue #2 is the interesting one: since the files we were creating were described in the %files section of the old package but not in the new one, the step 5 in the upgrade flow removes them. This, in my opinion, is a bug: the mtime of the candidate files for removal should be checked to make sure no scriptlet touched it.

The Tasty Bite

Enough of issues, let’s take a look at a situation where this upgrade behavior actually helped us solve a production bug:

In our first release some of our code depended on some filesystem data to work and we didn’t cache such data for further use throughout the service lifetime; This caused a bug during an upgrade because the newer package version would replace some files which the old service depended on - so, after an upgrade through our UI, the application would crash since the filesystem data is different from what it expected (yeah, I’m talking about templates - shame on us).

Such an issue could be solved by simply releasing a new minor version of the product, however the bug would still show up if the user tried to upgrade from the original version, without going through the minor update, so this wasn’t perfectly safe.

By the time we caught this problem we were already aware of the real RPM upgrade flow, so we used that to work around the issue for any version: in the %pre section of the new package we backed-up the old files which triggered the crash when changed and kept them at their original location after the upgrade finished - when the application restarts, it detects the presence of the old files and finally replaces them with the new ones: now the service is running with updated code aware of the filesystem changes and nothing really breaks.

This is what one would call a really ugly hack in the wild and I’d have to agree. However, the code was already in production and even worse than doing dirty hacks is letting the application crash in the face of the user.

Neat App of the Day: Taskwarrior

2011-01-22T00:00:00+01:00

Introduction

Taskwarrior isn’t an application, it is a project. To quote from their website:

Taskwarrior is an ambitious project to supercharge task with an interactive interface, GTD features, color themes, data synch, dependencies, custom reports, charts, and Lua plugins, all while our international team provides excellent support!

So, yeah, the neat app I’m talking about is actually called task, but that’s too hard to differentiate from anything else, since it’s a dictionary word.

Anyway, I’ve been searching for a fast and easy to use TODO-list manager for a long time in order to move away from that dreaded-but-functional TODO.txt plain-text file in my homedir. And for the past month or so, I could say I’ve finally found a decent replacement.

The Good

It’s a command-line app
High programmability level
Command abbreviation
Batch processing
Active development community
On-demand eye candy

Being able to quickly manage my tasks is a must-have for me, and by “quickly” I mean: without the need to use a mouse. Taskwarrior does just that, with added support for synchronization, tags, projects and calendaring. I can simply list my tasks for the day with task ls due:today (or the overdue ones with task overdue) and filter any kind of query command by project (pro:<NAME>), priority (pri:<H,M,L,N>), tags (+<TAG>) and many other modifiers (check the ATTRIBUTES AND METADATA section in the task manpage).

I could write several paragraphs about how many interesting features Taskwarrior has, but I’ll direct you to their official tutorials instead. If you prefer watching over reading, make sure you check this introductory video out:

The Bad

It’s only a command-line app

Being a command-line-only application, it’s usage becomes really limited and it’s quite impossible to do any sort of productive task management on a mobile device without a (proper) physical keyboard. Luckily this is bound to change (otherwise this would be placed under a “The Ugly” section) since there are plans to decouple the UI from the engine (I believe this’ll be on the next release: 1.9.4), which will allow lots of front-ends to be created, which is great for I’ll be able to develop apps for all those cool smartphones and gadgets that I don’t have (the prices are plain stupid here in Brazil).

Alternatives

There are plenty of TODO-list managers out there, some with really interesting features and great visuals. If having your list accessible off-line in an easy way is not a requirement, there are several web-based tools out there to try out - I’ve used Google Tasks for a couple of months and it’s quite simple (maybe too simple), a featureful alternative to Google’s take on task management would be Remember the Milk. If GUI applications are your thing, maybe Evernote or Getting Things Gnome will be more to your liking.

But if you’re like me and utterly dislike using your mouse, but Taskwarrior didn’t suit your taste, another great option would be the more popular todo.txt, which has the advantage (not for long, I guess) of having mobile clients for IPhone and Android.

Conclusion

At long last a task manager I can use without feeling uncomfortable or limited! I’ve been using Taskwarrior daily, coupled with some sup mail hooks to trigger task creation and syncing frequently with my personal tasks stored on Dropbox and my work-related ones stored on my current job’s cloud-like infrastructure. Make sure you try this one out if you’re not completely satisfied with your current solution, even if you tend to prefer GUI apps.

Neat App of the Day: Sup Mail

2011-01-11T00:00:00+01:00

One of the things I really love about open source is that one can find new amazing software often and quite easily. I mean real quality software, not that shitload of crapware/adware that’s usually associated with “free” software.

I’m well know for using “alternative” apps, for changing (technical) opinions quite often and for breaking my system a lot (it figures, I have a LFS, Gentoo and - currently -Archlinux background). Turns out that this beta (or rather, alpha) nature of mine gives me the opportunity to try some promising software out way before it gets widely known.

Introduction

This time I’ll write about an application I’ve been using since early 2010: sup. Sup is an e-mail client that works in the command line and has several neat features hardly found on major clients out there. It’s already well known by the bleeding edgers, but surprisingly unknown by most of the nerd/geek/whatever circles I know.

The Good

Written in Ruby
It’s FAST
Full-text search
Local storage
Scriptable
Tags!

If you are a gmail user, you should be aware of how much tagging helps organizing the flood of e-mails we receive every day. I use sup to deal with a couple of personal accounts and - mostly - with my job’s e-mail and I couldn’t be happier. Being able to search my e-mail using a gmail-like query language and get results in less than a second really helps out when you need to find that message you thought no one would care, or that meeting agenda you forgot to store in a better place.

Being written in Ruby really eases the creation of hooks for several events and, since I have some experience with the language, even allowed me to hack on the code and create a useful feature for me: contact groups (check my code).

The Bad

Written in Ruby
Needs scripts for pretty basic things
No (official) maildir sync-back support

You might be wondering how come being written in Ruby is listed here if it was listed in the section above. Well, if you’ve ever maintained anything in Ruby long enough you’ll probably know how things can break badly from version to version (even minors!). Using a bleeding-edge distribution doesn’t help either: most of the packages are either “too new” or just don’t work at all. Ever since Archlinux changed it’s default Ruby version to the 1.9.x family, running sup became a pain so, to avoid any problems, I have to keep an exclusive rvm setup for it and freeze updates on xapian-core.

Alternatives

There are several console-based e-mail clients out there, but none with the power of sup. If getting over those Ruby issues or using a relatively young software bothers you (and you can live without tags), it might be worth checking out the most well-known and established MUA for the Linux terminal: mutt. Other than that, only vmail comes to mind, which seems a good enough solution if you only use gmail (you gotta be a vim addict too, but who isn’t :-).

Conclusion

Tired of those sluggish GUI-based e-mail clients out there? Can’t find anything that fits your needs? Even grew tired of Sylpheed? Check out sup. If you manage to install it in your distro, I’m sure you’ll like it.

Custom Directory Completion for Zsh

2010-12-26T00:00:00+01:00

I’ve been planing on checking zsh out ever since I found out about oh-my-zsh. This weekend I finally had a little free time to do so. Besides a pretty informative prompt and useful command completion, one thing that I (now) find absolutely necessary in any shell I use daily it my custom make_dir_complete function to generate shortcuts for any directory. I’ve previously described how to implement it here, if you’re interested.

Well, given zsh’s fame of having a really complicated completion system, I was reluctant to go down the path of implementing it on my own and decided to check on #zsh (at Freenode) if anyone had already done what I wanted or if I could at least get some decent directions (zsh’s completion documentation is exactly like bash’s. They both suck). Since the only answer I got was in the form of ”I’ve asked that previously and got no answer”, I decided to give it a shot.

I went down the same chain of thought I had to use on bash: generate a completion function which would reply the next possible directories based on the first parameter. Turns out this was a really bad choice, since I soon got stuck at not getting consecutive completions and started feeling confused by the many glob options available.

After messing around with existing system completions (basic ones, like the man completion), I found some really interesting uses of compctl and decided to delve deep into it’s manpages and soon I came up with this:

function jk() { cd ~/src/$@; };
compctl -/ -W ~/src jk

And that’s it, it worked! I was impressed, even completing a directory with spaces worked out of the box. Now I just had to add the code generation part and would be all set:

make_dir_complete() {
    local aliasname=$1
    local dirname=$(readlink -f $2)
    FUNC="$aliasname () { cd $dirname/\$@ }"
    eval $FUNC
    compctl -/ -W $dirname $aliasname
}

Short, concise and way easier to “get it” than it’s bash’s counterpart. I’ve even created a oh-my-zsh plugin for using it, it’s the same function, but with proper usage documentation and parameter verifications. You can check it here.

So, all my requirements were met and I was ready to ditch bash in favor of zsh, right? Unfortunately no. I’ve spent the Sunday using it, was quite happy and everything was going smooth, until I decided to do a test-drive and check how it fared against my current job workflow, which involves a slightly large git repository (migrated from svn by yours truly). That’s when I came back running to bash. A great part of my job involves reviewing code, and the shell completion helps me a lot matching the branches I’ll review and merge and zsh’s git completion crawls like a zombie (a slow one, without legs).

A fully migrated setup, with every important thing working properly, discarded without remorse. I’m not willing to change shells and lose productivity, even if the community behind oh-my-zsh is big and amazing. So long zsh, I’ll be back (soon, I hope) when your git completion doesn’t take more time to list possibilities than it takes me to remember and type a branch name.

Custom Directory Completion for Bash

2010-10-10T00:00:00+02:00

UPDATE: Z-Shell users, check out the post detailing the zsh version

I really dislike writing tildes. I mean, since I use an international keyboard layout (In Portuguese we do some heavy usage of tilde and cedilla), I have to type <SHIFT>~<SPACE> in order to get a single ~ character.

For that reason, I’ve always used quick commands to change to directories nested from my homedir: first I used aliases, then some pushd/popd tickery and finally some proper bash completion usage. Up ‘till a few weeks ago, I’d only used a single command, which would complete the folders inside ~/src/ - but this has changed. I felt the need to have quicker access do my downloads folder (~/download/) and even for my homedir, so I hacked together a simple yet useful completion generator function which I though it might be worthwhile to share.

Here it is:

make_dir_complete() {
    local aliasname=$1
    local dirname=$(readlink -f $2)
    local prgname="__s_${aliasname}__"
    FUNC="function $prgname() {
        local cur len wrkdir;
        local IFS=\$'\\n'
        wrkdir=\"$dirname\"
        cur=\${COMP_WORDS[COMP_CWORD]};
        len=\$((\${#wrkdir} + 2));
        COMPREPLY=( \$(compgen -S/ -d \$wrkdir/\$cur| cut -b \$len-) );
    }"
    ALIAS="$aliasname () { cd \"$dirname/\$*\"; }"
    eval $FUNC
    eval $ALIAS
    complete -o nospace -F $prgname $aliasname
}

The usage is simple: make_dir_complete <shortcut_name> <directory>. So, in my case I’d call make_dir_complete jk ~/src and whenever I type jk <TAB><TAB> it will list to me the directories inside ~/src. More advanced completion like wildcard and partial matching (no fuzzy matching, though. Sorry) also work, thanks to bash’s awesome complete function.

There are probably better ways to achieve what I just did, so please let me know if you see any room for improvements :)

An IRC Bot for Managing Shared Machines

2010-09-25T00:00:00+02:00

This Saturday afternoon I had nothing (that I deemed) important to do, so I’ve decided to check how rusty I had became with Ruby. I’ve spent some fun hours coding a simple IRC bot to substitute the one that my team used at work since we’ve lost it’s source code.

There’s no technical reason for my decision of using Ruby instead of my current favorite programming language. I just wanted to practice with another language for a change, but wasn’t with enough disposition to learn a new one. I also had to bear in mind that (if used), this would be installed on a production machine, so keeping the external dependencies limited would be a good idea ¹.

Anyway, you can check it out here. I’ve included a basic README file with usage instructions. The fun part is over, so I don’t plan on documenting nor creating help / debug output messages today.

Using it is pretty straight-forward if you’re familiarized with cinch. If not, just check their homepage. The good thing is that I’ve done MachineBot (yeah, the name is lame, I agree) as a plugin, so if you already have your own bot built with cinch, integrating mine is just a matter of requiring it and adding to the plugin list.

I should really start doing this more often, it’s a great exercise. Too bad weekends like this are hard to come by nowadays. I’ll make sure to keep track of how much time I spend doing it next time - It’s been around 4 hours, but I’d like to know precisely what’s my pace.

Actually, I hadn’t thought of that while coding. This came to my mind while writing this post.
↩

Disgusted With C++ Build Errors

2010-08-07T00:00:00+02:00

Everybody knows C++ build errors are one of the most bizarre anyone will ever find in the compiled languages world and, by knowing that, it seems that the developers treat new introduced oddities as a normal thing.

Don’t get me wrong, I’m not a C++ hater ¹ and I didn’t come here to rant about the well known and blatantly ugly compilation errors caused by any kind of incorrect Template and/or Overloading usage. What made me come home and write this post was this question on Stackoverflow.com.

Just reading the question and it’s answer is enough to get sick. This is basically what goes on:

A fella decides to try out C++0x’s standard multithreading support and uses the following code:

#include <iostream>
#include <thread>

void hello() {
    std::cout << "Hello Concurrent World";
}

int main(int argc, char *argv[]) {
    std::thread t(hello);
    t.join();
}

He says he compiled it with g++ -std=c++0x -g -o executable source.cpp and there were no errors, but when running he got a weird Segmentation Fault:

Program received signal SIGSEGV, Segmentation fault.
_dl_fixup (l=0x7ffff7b0992c, reloc_arg=<value optimized out>) at ../elf/dl-runtime.c:147
147     ../elf/dl-runtime.c: No such file or directory.
        in ../elf/dl-runtime.c

When trying his code on my box, I got this:

terminate called after throwing an instance of 'std::system_error'
  what():
Aborted

Which is not as ugly, but still very weird and non informative. Examining the backtrace we get not even a single decent hint about what may be going on:

0xb7fe1424 in __kernel_vsyscall ()
0xb7d67d61 in raise () from /lib/libc.so.6
0xb7d695ee in abort () from /lib/libc.so.6
0xb7f7bcd5 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
0xb7f79bd5 in ?? () from /usr/lib/libstdc++.so.6
0xb7f79c12 in std::terminate() () from /usr/lib/libstdc++.so.6
0xb7f79d81 in __cxa_throw () from /usr/lib/libstdc++.so.6
0xb7f1eee9 in std::__throw_system_error(int) () from /usr/lib/libstdc++.so.6
0xb7f721d0 in std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>) ()
 from /usr/lib/libstdc++.so.6
0x08048dd2 in thread<void (*)()> (this=0xbffff53c, __f=0x8048b0f <hello()>)
 at /usr/lib/gcc/i686-pc-linux-gnu/4.5.0/../../../../include/c++/4.5.0/thread:135

Now, if you’re even a little bit used to programming with threads on C or C++ you might just have noted that there’s no flag being given to g++ to link the binary against a threading library (pthread, for example, on Linux) and that’s exactly the issue here. He should have compiled his code with g++ -std=c++0x -g -pthread -o executable source.cpp.

How fucked up is that? I mean, it’s logical that one should need to link against a threading library (even though I believe that this should be automatically resolved by the compiler, being part of the standard and such), but no error at all while compiling? Wtf? That’s not an unauthorized access to private memory, null-pointer dereference nor anything like that, it’s a linking issue and must cause an error no matter what.

I’m obviously being too harsh with all this. C++0x is not a released standard yet and g++’s implementation is based on the draft, which is constantly (not that much anymore) changing, so perhaps there was not enough time/interest to focus on these “usability” issues. I just hope this behavior changes. Soon.

Though, if given the choice and it sounds reasonable, I’d definitely stick to plain C most of the cases. I’d rather glue Python and C together if the only argument for using C++ is being able to “properly” use OOP.
↩

Splitting Git Repositories

2009-12-14T00:00:00+01:00

Today I was doing some maintenance on my repositories and decided some of them were worth splitting into sub-projects. Since I couldn’t find any information on Google on how to do it the easy way, I dug up a method myself, which I’ll describe bellow.

Background

Let’s say you have a git repository in which you store some toy codes, I’ll call it playground.git. If you keep it long enough, soon you’ll see some of your toys growing too large and/or important to keep on a mixed messy repository.

For illustration purposes, I’ll assume your repository is structured this way:

.
`-- playground.git/
    |-- subproject1/
    |   `-- ...
    |-- subproject2/
    |   `-- ...
    `-- important_project/
        |-- important_file
        `-- ...

What we’ll see here is how to migrate the folder important_project to it’s own repository (say, new_project.git).

Chances are you’re sharp enough to notice when a project is growing too important to be on your playground, so moving it away by simply copying shouldn’t be such a loss, but if you find it’s important to keep the commit history and/or just noticed your project was getting bigger several hours of work later, you may find this post useful.

Finding the Correct Revision Number

This may be quite obvious to some of you, but it’s a rather useful command (flag, actually) which I don’t see many people I know using.

From git-log manpage:

git log [<options>] [<since>..<until>] [[--] <path>...]

The useful parameter for our case is “path”, so if we want to see the logs pertaining to a given file/folder, we’d use:

git log -- important_project

So, basically, if we want to find the first commit that happened inside our important_project folder, we could issue the following:

git log --oneline -- important_project | tail -n 1

Which would output a shortened commit-id followed by the commit message. What we want here is the commit-id, let’s say it is DEADBEEF.

Now, if we do a git diff DEADBEEF we’ll see the changes since the commit-id DEADBEEF. Since we’re migrating everything inside important_project we actually want to include the first commit, so we’ll be using DEADBEEF^, which points to the commit before the one we found.

Generating the Patches

Now comes the part where git shines: remember the -- <path> option we used before? It’s not explicit on git-format-patch manpage, but you can use it here too! So creating patches is simply:

git format-patch DEADBEEF^ -o ~/Desktop --relative -- important_project

Let’s understand what’s going on here:

The -o ~/Desktop option is telling git to write the patches on my Desktop folder. I like to store them there so I don’t forget to work with them later on.
The --relative switch is not really useful with this example, but it’s useful if you’re deeply nested inside your repository. This switch will instruct git to not include the full path (from the project’s root, of course), but to use your relative path.

See the manpage for git-format-patch for some other useful options.

Creating the New Repository

You’ve probably done this before, so I’ll be quick:

mkdir new_project.git
cd new_project.git
git init

Applying the Patches

This part may get tricky (to understand, that is) if you’ve generated patches for projects too nested on playground.git.

If you’ve managed a simple git repository before accepting incoming patches from third-parties, you may know the git am command. That’s what we’ll use, but with a little switch the people coming from the dark ages of VCSes probably know: the -p<NUM> for relative patching.

For you fellas who didn’t have to mess with the patch command by hand ‘till now, a basic explanation of the -p<NUM> option would be: the number of “folders” to go up when applying a patch, so, say a patch diffs the file “/important_project/file.txt” but you actually want to apply to “./file.txt”, you’d use -p2.

Okay, time for the magic:

git am -p2 ~/Desktop/*.patch

And that’s it. I’m impressed with git. Again.

Watch out when using Python's urljoin

2009-10-24T00:00:00+02:00

While implementing some pinging functions I’ve come around a weird behavior of the urljoin method from urlparse:

# weird_urljoin.py file
from urlparse import urljoin
print urljoin('http://url.com/pathname', '?q=joinable')

What do you think will be printed when this code is executed? Whatever you do, you probably wouldn’t think the following would happen:

python weird_urjoin.py
"http://url.com/?q=joinable"

Yeah, while testing if the urls would hit I was building them with urljoin and getting a 400 status code every request without knowing why - I was actually printing the url I was trying to hit, but the change was so subtle that I didn’t notice.

Turns out that this behavior is expected, since the function clearly says it implements the rfc1808 standard (and actually implements the rfc2396, if we check the implementation).

Anyway, if you check the RFC I’ve linked above, you will note that they’ve both been obsoleted by rfc3986. So I decided to see if this was “fixed” in a newer python version ¹, and well, it was! (if you’re going to check it yourself, note that urljoin is now part of urllib.parse, which makes a lot more sense).

After a quick investigation, I came across Issue 1432, which basically goes over all I’ve just written here.

If you’re a Python 2.6 user, you may be inclined to say that urljoin('http://url.com/pathname', '?q=joinable') works alright and it does. On 2.6. On Python 2.5 the result is the same as my first example. Check out this simple test and it’s output:

try:
    from urlparse import urljoin
except ImportError:
    from urllib.parse import urljoin

print(urljoin('http://url.com/pathname', '?q=joinable'))
print(urljoin('http://url.com/pathname?', 'q=joinable'))

# On Python 2.5.4:
# >>> http://url.com/?q=joinable
# >>> http://url.com/q=joinable
# On Python 2.6.3:
# >>> http://url.com/pathname?q=joinable
# >>> http://url.com/q=joinable
# On Python 3.1.1: "http://url.com/q=joinable"
# >>> http://url.com/pathname?q=joinable
# >>> http://url.com/q=joinable

The behavior is the same with 2.6 and 3.1, but on 2.5 it get’s slightly more messed up. It’d be swell if the fix from Issue 1432 got backported to 2.5 or even better: AppEngine started working based on 2.6, so I’d finally stop using 2.5. There are some great improvements that I’d love to use here.

Okay, what to do if I’m developing for 2.6 but want to keep compatibility with 2.5? I’d just use string formatting to join the urls and the parameters. Actually, I think that’s the best approach even when not keeping compatibility with older versions:

endpoint = '%(url)s?%(data)s' % data_dict

Seems way more readable to me than:

endpoint = urljoin(url, '?%(data)s' % data_dict)

And we don’t need to care whether the url is fully-qualified. So, concluding this article, I’m giving up one of my advices posted previously, at lest when doing quick coding, since you must pay great attention when using urljoin.

Even with all these considerations, there’s one thing you must always do: test. A LOT.

I’ve been messing with Python 3k for a long while now and whenever I have to do some quick tests in the interactive console, I go for Python3 instead of Python 2.5, which is used on AppEngine. This is sub-optimal, I know, but I get to train my mind for the changes being introduced by the new python version.
↩

Markdown Update

2009-10-18T00:00:00+02:00

If you’re following the commit log for this blog engine, you’ve probably noted that I’ve just switched from python-markdown2 to the original python-markdown.

This is a change that I’ve already discussed with Victor, who is a user from the old version of this engine and has been helping me on this one with occasional commits. The main reason for the switch is extendability. Extending python-markdown is way easier, and it also comes with a few useful built-in extensions with aren’t, AFAIK, available on python-markdown2:

Abbreviations (I’ve just used it! :-) tip: It’s usually in uppercase)
Definition Lists
Tables
Table of Contents (this one is additional, not included in the “extra” package)

Aside from the extensions provided, it’s on my plans creating a few hooks to ease the usage of this blog, such as:

Image tag for uploaded images, without the need to provide the full relative path
LaTeX rendering (I’ll leave this one for Victor, since he’s the one that suggested it :-)
Anything that comes to mind

I’ve been seeing an increased usage of python-markdown2 in open source projects, and I think it’s mostly based on it’s advertised speed compared to the original, and on the fact that it’s a simple .py file. Actually, I wouldn’t advise against it – It’s just that I need to be able to extend it. Also, if you’re interested in a comparison between both libraries, I’d suggest reading Reviewing Markdown by Greg Wilson.

Oh, and you may have noted already, but let me say it officially: syntax highlight with pygments was finally enabled.

Ditching PyTz

2009-10-11T02:00:00+02:00

I’ve just removed pytz from this blog.

The Reasoning

There are a couple of reasons I’d like to state:

It is too big.
Loading the module takes too long.

You may say say we can get around the first issue (taking into account AppEngine’s 1000-file-limitation) by zipping the zoneinfo directory and patching the open_resource function (since the old workaround doesn’t work anymore). Doing so brings us to the second issue.

On my development server, hitting the homepage for the first time would take a few more seconds than what I like to wait when testing small changes and checking the logs it was clear that it got stuck loading the zoneinfo.zip file. Also, checking the dashboard for this blog one could see some high cputime raises from time to time and my cpu consumption was growing much faster than expected.

The Solution

Given one of the sayings from software development features best practices: “Never take away a feature. Ever. Even if your users don’t use it.”, I’ve started browsing for lighter and cheaper alternatives.

Not long later, I find one of those useful services which I never heard about: json-time, which given a timezone, returns it’s current time. And that’s decided! Took me a couple of minutes to find out about Django’s simplejson, which is available on AppEngine.

The Conclusion

Well, if you check json-time source code, you’ll see that it is a simple service based on pytz, so one can say that I’ve not completely ditched pytz and actually, I was hoping I wouldn’t.

Pytz still rocks, but it would be really nice if it was included on AppEngine’s environment (or a library with the same functionality was added).

How to read JSON data from an AppEngine app

2009-10-11T12:00:00+02:00

UPDATE: I’ve removed the urljoin function usage since it does not work as expected.

Since I had to figure it out myself, I may as well write it here, so it may help somebody else some day.

In this post I’ll write how to fetch and decode json data in an AppEngine application. It is really simple and involves using two modules: simplejson and urlfetch.

from django.utils import simplejson
from google.appengine.api import urlfetch

Following best practices, and to avoid issues with joining two urls, we’ll use the urljoin method from urlparse and urlencode from urllib and enable logging:

import logging
# urljoin follows rfc1808 and doesn't work as a human would expect
# from urlparse import urljoin
from urllib import urlencode

For this example, I’ll use the json-time service cited on my previous post, but it should work with any service which returns json data as content for requests.

SERVICE_ENDPOINT = 'http://json-time.appspot.com/time.json'

To use json-time, you must pass a tz element as GET parameter, like tz=America/Sao_Paulo. So let’s build our url:

encoded_data = urlencode({'tz': 'America/Sao_Paulo'})
# url = urljoin(SERVICE_ENDPOINT, encoded_data)
url = '%s?%s' % (SERVICE_ENDPOINT, encoded_data)

You might be wondering why not just write the full url (http://json-time.appspot.com/time.json?tz=America/Sao_Paulo) instead of messing around with all those libs. Well, in this case, it’d really be easier, but consider a more complex service with many options, or one which would return the options after a first request. Using those modules makes it all more flexible and less human-error prone.

Now we’ll use google’s urlfetch to fetch our properly built url and check it’s results:

result = urlfetch.fetch(url)
if result.status_code != 200
    # Boo! It didn't work
    logging.error('json-time returned unexpected status code: %d'
                % result.status_code)

Now we’ll create a JSON decoder and try to read the data:

else:
    received_content = result.content
    json_decoder = simplejson.decoder.JSONDecoder()
    try:
        decoded_json = json_decoder.decode(received_content)
    except ValueError:
        logging.error("json-time sent data which simplejson couldn't parse")
    else:
        logging.debug('Decoded JSON from json-time: %s' % str(decoded_json))

And that’s it! You’ll see on your app’s console whether things worked alright or not ¹. If it all went well, you’ll be able to verify that decoded_json is simply a dict like the following:

decoded_json = {
    "tz": "America\/Sao_Paulo",
    "hour": 1,
    "datetime": "Sun, 11 Oct 2009 01:42:48 -0200",
    "second": 48,
    "error": false,
    "minute": 42
}

Using it is quite straigtforward and is completelly up to you. Hope it helped :)

I didn’t really test this whole code in production, but I’m using a variation of it for this blog - you can check it here, in the tz_date filter.
↩

Bazaar Weirdness

2009-07-24T00:00:00+02:00

As you know (or at least should, if you read this blog), part of my work on Summer of Code involves hacking around Ubuntu’s Apport.

I’ve just finished implementing the credentials API I’ve submitted earlier (another branch merged, yay!) on the Bugzilla back-end and found myself with a weird issue related to the repository format. I didn’t delve much on it, but it seems the upstream branch has migrated to a new format (to support bzr-svn and bzr-git, I guess) and now I have to migrate my branch in order to merge the changes from mainstream.

This kind of issue is a real turn-down for me. I may be doing something wrong, but in my opinion this should never happen, even more with Bazaar, which - like Mercurial - advocates a “sane” UI. I’ve never had a single issue merging/rebasing/branching using either git nor hg, and I’ve been using both for quite a while now.

Nevertheless, this post is just a quick rant. I’ve issued bzr upgrade --rich-root-pack /path/to/remote/branch and it’s already transferred 60MiB. For a format upgrade. And there is no option to disable the backup “feature”. How fucked up is that?

I’m about to abort the upgrade and create a new branch, ignoring the historical value of the prior commit messages.

Update: The upgrade finished after \~75MiB (It erased the line when completed, so I didn’t see the final value). I’m just too afraid to do a bzr get on my branch now to check if this upgrade reflected on the repository size.

Update: I’ve just run a bzr get and the download size was \~6.2MiB more than three times what it used to be.

GSoC Midterm Roundup

2009-07-13T00:00:00+02:00

That’s it. The midterm evaluation week is gone (actually, there are a few hours left, but it’s gone for me, at least).

I’m writing this up while I wait for my Univ’s restaurant to open and it’s quite cold here - I’d usually like it if it weren’t for my lack of sleep (too-much-coffe), I guess.

So far I’ve been on schedule, even though I feel like I could be way ahead of it if it weren’t for some unexpected issues that came up.

Well, the Bugzilla back-end has progressed little since last week: it’s feature complete, but it needs a log-in mechanism which is on the way as I discuss with Martin Pitt the best way to do it. PyBugz is going steady, I’ve a new feature for reading cookies from the most common browsers (Firefox and Chrome at the time being) uncomitted yet - What worries me is the lack of response from PyBugz maintainer/author.

On to following my schedule for the last term, I’ve began mocking up (on paper) some user interfaces for the stand-alone bug reporting tool, which will probably use PyBugz and maybe fire-up apport when the user is reporting an application crash. I still haven’t decided which way to go: power or simplicity. The best would be to put both together but it is quite hard to think of an intuitive user interface being a command-line addict myself.

As always, I’ll end promising to post more frequently, but I can’t promise to keep my promise :)

GSoC Project Status

2009-06-19T00:00:00+02:00

I haven’t been using this blog to report my status on Summer of Code. Well, actually, I haven’t been using this blog at all.

So, um, okay… I’ll write what I’ve been doing. That’ll basically be a modified status report I sent to my mentor sooner.

The Bugzilla back-end is finally feature complete! Every test (ported from Launchpad back-end) is passing when testing with 3.0 and 3.2 Bugzilla instances.
To integrate Apport on Pardus, we’re gonna need a PiSi back-end implementation. Luckily, PiSi is written in python and it’s interface is quite well defined, so I already have a partially working version. Finishing it shoudn’t take as long as the Bugzilla back-end took.
To correctly act when requested, I had to fork PyBugz fixing some issues (check the revision history, if interested) and implementing a new feature to modify attachments. The reason I forked it was due to the lack of reply from PyBugz original author - but since his branch is on GitHub too, I’ve made a pull request, so my changes can get upstream.

Talking about upstream, while testing the Bugzilla back-end, I’ve talked to apport’s maintainer regarding a python-apt API change issue, which culminated in one of my branches being merged upstream. So that’s actually my first real code contribution to a successfull open source project. Cool, huh? :)

Sobre Brigas e Pirraças

2009-06-05T00:00:00+02:00

I’d like to apologise to my English readers. The following post is completely in Portuguese (from Brazil) and I won’t bother translating it since it’s about issues in this country. Perhaps an automated translation will help you if you’re really interested.

É comum ouvir em meio acadêmico que a “geração coca-cola” é mais acomodada, não luta por seus direitos e é extremamente individualista. Culpa-se, claro, nossa herança maldita, o meio em que nos desenvolvemos, a espelho de uma cultura extremamente consumista e apática.

Apesar de acreditar que há sim uma parcela de “culpa” na desculturalização que vem ocorrendo, culpar somente isso, muitas vezes com algumas pitadas de teoria da conspiração, é errôneo.

A massa de descontentes é notável. Basta visitar fóruns on-line sobre assuntos diversos que sempre há uma série de mensagens de insatisfação política - muitas delas pessimamente argumentadas e/ou infundadas, de fato, porém não é isso que importa. Saindo do meio da Internet, é só conversar com qualquer desconhecido que logo surge um assunto sobre o governo e o quanto ele está ruim ultimamente - não importa o ano nem a pessoa que está no poder.

A “geração atual” perde muito seu tempo discutindo os problemas nacionais em fóruns de discussão, assinando petições on-line, encaminhando e-mails esclarecedores a seus amigos. Então, por quê nada acontece?

A revolta e a revolução são vítimas da modernização. Os meios de comunicação evoluíram, mas as formas de fazer-se ouvir quando o alvo pouco se importa continua a mesma. Organiza-se um FlashMob com centenas de pessoas em instantes, mas parece impossível reunir estudantes indignados para protestar contra o aumento absurdo das tarifas de Ônibus.

Não é culpando a internet que faremos os gordinhos tetudos (afinal, quem frequenta fóruns on-line só pode ser gordo, não é mesmo?) levantarem-se e brigarem por seus direitos. A mídia mudou, os interesses mudaram, será que não é hora de o recrutamento mudar? - o que está enferrujado é a “massa revolucionária”, a força estudantil, que se precariza a cada ano com o excesso de politicagem, inúmeros jogos de interesse e influência partidária, que luta contra o “imperialismo norte-americano”, mas nem olha ao seu redor para ver o chiqueiro em que estão fazendo todo alvoroço.

Deve-se lembrar que não é preciso concordar em tudo para que se lute em conjunto por uma causa e que hoje em dia temos acesso a um volume de informação absurdo, de forma que torna-se difícil confiar em qualquer organização. Não se briga por um plano completo, por uma pauta política, por um partido. Luta-se por mudança.

Não entrarei nos (poucos) méritos da democracia, pois é uma área perigosa e cheia de pessoas com facas nas mãos e gigantescos dedos indicadores para lhe apontarem e rotularem com algum substantivo qualquer. O fato é que, no meio em que vivemos, jamais haverá uma idéia em uníssono, portanto reitero: lute por mudança, não por um ideal completo de quinhentas páginas. Se a mudança só piorou as coisas, lute novamente.

Estamos recheados de derrotismo e egoísmo. Não importa de quem é a culpa, mas sim como e quando vamos mudar isso.

Branch Published!

2009-05-24T00:00:00+02:00

As promissed: I’ve just published my branch from apport on Launchpad, check it here.

Since I wasn’t working directly on a bazaar branch, my first commit was quite a big blob (and I even forgot to pull from uptream before publishing) - I’ll try to keep the next ones short and concise.

A new task has appeared when I started coding the Bugzilla crash database for apport: I’ll need to write a a PiSi backend for it to query for package info. There’s already a preliminary working version of it thanks to the amazingly well-written and well-organized PiSi source code.

Not much more to write about right now - The coding phase has officially stated but due to this new task I’ve described, I’m way behind what I expected, even though I’m a bit advanced if compared with my official-and-private tasklist.

As a reference: most of my commits will probably flow on weekends, tuesdays and fridays since that’s when I have a little extra free time ;)

Coding phase: START!

2009-05-14T00:00:00+02:00

Actually, the official coding phase is scheduled to start on 23rd, but I decided to start a bit earlier since I don’t have much free time.

We’ve finally decided to go with Apport and somehow follow the steps from my original proposal. So now I’ll be officially using three of the most well-known DVCSs: git, mercurial and now Bazaar.

The initial work is fairly simple since Apport is quite well modularized, so I expect to finish it before the oficial “Coding Phase” starts. As usual, when I decided to start working some problems started showing up (exams and projects scheduled for next week, new assignments, the flu), so all I could do up untill now was set-up a testing environment, play around with python-bugzilla functionalities and get to understand how apport will manage multiple crashdb implementations.

I have yet to lean how to properly contribute using Bazaar, but it seems to be fairly easy. So, whenever I set-up a branch with my code I’ll update here.

On Summer of Code

2009-04-27T00:00:00+02:00

… finally!

The reason that made me try blogging again was my acceptance as a student on Summer of Code, so I guess I should be talking about it here.

I’ll be working with Pardus folks with a project entitled “Bug Reporting Tool for KDE4”. I’ll post the original proposal here, but bear in mind that we’re in the middle of a discussion about what path to take given KDE’s drKonqi current state.

Project Abstract

A KDE 4 application to help non-technical users diagnose and report bugs and issues. This application will provide the following: a simple template-based bug-reporting assistant, a GUI front-end for crash reporting on Bugzilla and a “Hardware Test” wizard to check basic computer functionality.

There are two notable projects that strive this functionality: Apport and abrt. This project’s goal is to improve on the usability and portability of either and further extend it to fill Pardus’s view of the issue.

Project Detailed Description

Based on the high-level description of the idea, this project would involve the following steps:

Decide which existing solution (Apport or abrt) is the most suitable.
Develop a Bugzilla back-end if necessary.
Define the simple Bug Report tool functionality: wizard-like interface or a simple dialog which would search for patterns on the text and suggest the inclusion of command dumps and configuration files (which would happen automatically).
Create a simple front-end for the chosen crash handler with passive or intrusive behavior depending on the severity and user-configuration.
Integrate the crash handler on Pardus.

Additionally, the whole proccess would be documented through a personal weblog and, if given the time, the creation of a “Hardware Testing” wizard would be studied and implemented.

Given the steps defined above and some superficial study of the available solutions, I’ve come up with some decisions (subject to changes after some discussion with the project mentor) which I enumerate:

Choose Apport over abrt: Apport seems to have a more mature code base, mostly due to have been deployed on Ubuntu for a long time; It’s modular interface and the fact that it’s coded in Python would greatly improve the data-handling and development process.
Develop the simple bug reporting tool first, mostly due to the fact that it would help testing the bugzilla backend.
Use python-bugzilla to develop a crashdb implementation of Bugzilla for Apport.

Even though I’ve attended to an Human-Computer Interaction course, I’m no usability expert, so the GUI-related parts would probably involve a bunch of mock-ups and at least one brainstorming session do decide which path to follow.