Hell Oh World!

Back in 2012 I wrote a patch to optimize ftell so that it doesn’t always flush buffers when it just has to report the current file offset. That patch unleashed a whole lot of pain due to the way in which the ftell code was written - it shared code with fseek, which is semantically different from ftell in a few ways. The general mess in libio code didn’t help matters much either.

Since that patch, a number of fixes went in to correct broken behaviour and it culminated in me essentially rewriting ftell. The main problems encountered were related to caching of the file offset in the underlying file to avoid a syscall, so Carlos suggested I write up a wiki document explaining the various scenarios. So I wrote File offsets in stdio stream and ftell in the glibc wiki.

Since this is a new ftell implementation, we’d love to get feedback on correctness (as bug reports) and performance (as bug reports, email, tweets, etc.).

We have been talking about having a patch review system in place for glibc for some time now since the volume of patches has been steadily increasing and we don’t have enough reviewers to go through them in time, leading to missed patches and general contributor unhappiness. Due to the fact that we’re a primarily email driven project, we needed something that fits into our current workflow and patchwork was the obvious choice to start with. I decided to do a setup on my domain first to get a feel of things before I made a request to set up a patchwork instance on sourceware. The instance is live now on patchwork.siddhesh.in for glibc contributors to get a feel of it.

The hard part about patchwork is that the documentation is a myth. There is an INSTALL file that sort of works, except that it doesn’t the moment you decide to use slightly different settings. Additionally, the instructions are targeted at dedicated hosting providers, so they almost completely don’t apply to someone trying to set up patchwork on Dreamhost on their shared hosting account. Of course, figuring out what to do with your patchwork installation once it is done is an adventure as well, since there seems to be no user documentation at all. Anyway, here’s how I did it:

Setting up the server and getting sources

I assume you have a shell account for the user that would be administering the subdomain, since you’d be doing a fair bit of sysadminy stuff on it. Also, it’s assumed that you’re hosted on a Linux based server; I don’t really care about how it works for Windows based hosting.

Create your Dreamhost subdomain using the control panel and make sure you have Passenger support enabled. The Passenger support creates a directory called public in your subdomain directory. Don’t bother setting up django at this stage.

Now get patchwork from their git repo:

$ git clone git://ozlabs.org/home/jk/git/patchwork

and copy the contents (i.e. whatever is inside patchwork, not the directory itself) into your subdomain directory. The contents of your subdomain directory would then be something like this:

$ ls
apps  docs  htdocs  lib  public  templates  tools

Now remove the public directory and create a symlink from htdocs.

$ ln -s htdocs public

We could technically just copy things over from htdocs to public, but it’s easier to update this way.

Next, we need django. patchwork needs django 1.5.x, so if your server doesn’t have it, you’ll need to download the sources yourself. To check the installed django version:

$ python
Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48) 
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> django.VERSION
(1, 2, 3, 'final', 0)
>>> 

Since we don’t have 1.5.x, we install django from sources in the lib/packages in your subdomain directory:

git clone https://github.com/django/django.git -b stable/1.5.x

Create a directory lib/python in your subdomain directory and symlink the django installation in it:

$ ln -s ../packages/django/django ./django

Configuring django and patchwork sources

The first thing to configure is the database. From your dreamhost control panel, create a mysql database and user. Have that information handy to put in your django/patchwork configuration.

The default settings for patchwork (and django) are in apps/settings.py. We need to override those by creating our own file called apps/local_settings.py. The first thing to go in our local_settings.py is our database configuration:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'db_name',
        'USER': 'db_username',
        'PASSWORD': 'super_secret_password',
        'HOST': 'mysql.myhost.name',
        'PORT': ''
    },
}

The instructions in the patchwork documentation mention using DATABASE_* style variables, but they didn’t work for me, probably because I figured out that I had an older version of django after I was done with the initial configuration.

Next you need set the following variables in local_settings.py:

SECRET_KEY = 'a random generated long string'

ADMINS = (
     ('Super Admin', 'super@foo.com'),
)

TIME_ZONE = 'Asia/Kolkata'
LANGUAGE_CODE = 'en-us'
DEFAULT_FROM_EMAIL = 'Patchwork (foo.com) '
NOTIFICATION_FROM_EMAIL = DEFAULT_FROM_EMAIL

# If you change the ROOT_DIR setting in your local_settings.py, you'll need to
# re-define the variables that use this (MEDIA_ROOT and TEMPLATE_DIRS) too.
ROOT_DIR = '/path/to/patchwork.foo.com'
TEMPLATE_DIRS = (
    # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
    os.path.join(ROOT_DIR, 'templates')
)
# Absolute path to the directory that holds media.
# Example: "/home/media/media.lawrence.com/"
MEDIA_ROOT = os.path.join(
    ROOT_DIR, 'lib', 'python', 'django', 'contrib', 'admin', 'media')

The SECRET_KEY can be generated using the following python snippet:

import string, random
chars = string.letters + string.digits + string.punctuation
print repr("".join([random.choice(chars) for i in range(0,50)]))

Other options are obvious from their names and values, so adjust them to your taste. ROOT_DIR is set to the directory where patchwork is, i.e. your subdomain directory. TEMPLATE_DIRS and MEDIA_ROOT are derived from ROOT_DIR, so I’ve just copied those over from settings.py.

Next up, we need to get static files for admin sessions into a place where django can find and serve it. They’re present in contrib/admin/static in your django installation and we need to copy them over to htdocs/static. Once this is done, we need to tell django that it can find the static files by adding the following configuration snippet to local_settings.py:

PROJECT_ROOT = os.path.normpath(os.path.dirname(__file__))
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'static')
STATIC_URL='/static/'
ADMIN_MEDIA_PREFIX='/static/admin/'

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.admin',
    'django.contrib.staticfiles',
    'patchwork',
)

Finally, we don’t want to spew out debugging messages to the server and we want to be able to debug problems at the same time, so we need logging support to be enabled. Add the following snippet to local_settings.py:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': '/path/to/patchwork.foo.com/django-debug.log',
        },
    },
    'loggers': {
        'django.request': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

making sure that the path in the ‘filename’ is writable by django. We can now disable debugging, so add this:

DEBUG=False

Now here’s the fun part, disabling debugging changes the behaviour of django, in that it suddenly starts doing extra checks, due to which you’ll start seeing failures like below:

SuspiciousOperation: Invalid HTTP_HOST header (you may need to set ALLOWED_HOSTS)

so fix this by adding the following:

ALLOWED_HOSTS = ['your.subdomain']

These are all the settings you would normally need to get patchwork up. Now we run manage.py from the apps directory (note that the instructions in INSTALL are slightly wrong here):

$ PYTHONPATH=../lib/python ./manage.py syncdb

This should initialize the database for patchwork and django. Follow whatever prompts that come up till you come back to the shell. If the command throws errors, read up and fix your configuration. The django documentation is surprisingly good once you get used to the layout, so don’t despair if patchwork documentation doesn’t help (it won’t).

Getting patchwork up

With the database set up and the sources in place, one needs to tell apache how to serve content through django. We had set up passenger for precisely this, so we just need to add a python script in our subdomain directory to tell passenger what to do. The script should be named passenger_wsgi.py and it’s contents should be:

import sys, os
basedir = os.getcwd()
sys.path.insert(0, os.path.join(basedir, 'lib/python'))
sys.path.append(basedir)
sys.path.append(os.path.join(basedir, 'apps'))
os.environ['DJANGO_SETTINGS_MODULE'] = "apps.settings"
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

This tells passenger where to find the app and what paths to use to find our custom django and any additional libraries and also our settings. At this point, browsing to your site should get you to a working patchwork installation. Login to the admin page (which should be http://patchwork.foo.com/admin/) with the superuser name and password you had created while running manage.py.

The first thing you’ll need to do is create a Site. Make sure you delete the example.com there since it will otherwise be used first and you won’t like it. After the site change, if you start getting a server error, then look at your django log and look for this error:

DoesNotExist: Site matching query does not exist.

This likely means that there was some data that referred to example.com and broke. You’ll have to edit the row for your subdomain in the django_sites table and change the id to 1.

Next, you add a project of your choice. It took me a while to understand what the names of the fields in the new project form meant, but maybe that was because it was getting very late. Anyway, Linkname is any name you wish to give to the project, which appears in the URL – I used glibc here. Name is a free form name for the project – I used GNU C Library. Listid is the List-ID header in the mailing list email that it should look for. Listemail is the email address for the mailing list. I just ignored the rest of the fields.

Getting emails from the project

Patchwork is useless without emails, so subscribe to the mailing list whose patches you want to monitor. You obviously need an email address for it, so create an email address first. There is usually a confirmation step involved in mailing list subscription, which should also be completed.

Next step is to forward emails from this account to the shell account where the site is running. To do this, dreamhost has a wiki document that describes how to link an email address to a shell account email. Follow that and send test emails to make sure that emails are reaching the shell user.

Now we need a mechanism to forward the mails received by the shell user to the parsing script in patchwork. This is easy to set with a forwarder file for postfix, called .forward.postfix in your home directory with the following contents:

"|/path/to/subdomain/apps/patchwork/bin/parsemail.sh"

Now you should start seeing patches in your project view on the website.

Sending emails from patchwork

This bit is fairly straightforward – patchwork needs to be able to send emails via SMTP for various events or for registration confirmation. Django needs the following information in local_settings.py to be able to send these emails, or else it uses the local SMTP, which won’t work on Dreamhost:

EMAIL_HOST = 'mail.patchwork.foo.com'
EMAIL_HOST_USER = 'from@patchwork.foo.com'
EMAIL_HOST_PASSWORD = 'supersecret'
EMAIL_USE_TLS = True

Getting started with patch reviews

When browsing, you’ll quickly figure out that you can’t really do anything with the patches you see there, unless you have sent a patch, i.e. your email address configured in patchwork is the same as the email address of the patch sender. This is not really very useful as far as peer reviews are concerned and working through the UI doesn’t tell you anything. One would obviously gravitate towards the admin UI to see if there are any settings there in the Users or Groups or Projects or People sections, but there are none.

The option is effectively hidden in plain sight in the User Profiles section. Click through to your user (or any user you're interested in updating) and come to the page that shows the projects that you're part of (Primary Projects) and projects that you're maintainer of (Maintainer Projects). The list of projects that show up next to Maintainer Projects aren't projects you're maintainer of, unless they're selected! So select the project(s) you are maintainer of and save your changes. Now when you login as that user, you'll see options to change patch review state and even delegate to other reviewers.

Final thoughts

It was quite an exhausting experience to get patchwork working and it wasn't just because it was on dreamhost. The project is fairly poorly documented and the usual clumsiness associated with web apps didn't help things. My relative inexperience with django may have compounded the problems I had getting this up, but I would again blame that on clumsy webapp syndrome.

I have written this down with the hope that someone else looking to do this would be able to get patchwork up in a bit less time than I took, and also because we may have to do it again on sourceware when there is consensus on using it. Given my wonderful memory, I'll probably end up making all the mistakes once again when I try it out the next time.

I have always had a peripheral interest in application security, so when some folks at work were discussing about attending nullCon (considered India’s premiere security conference), I decided to join them too. As usual, I submitted a talk because if selected, it pays for your attendance and it makes it easier to interact with more people.

I demoed and spoke about the recent pt_chown vulnerability in Linux and glibc, slides are here. Special thanks to Martin Carpenter for finding this vulnerability and later being available for advice and help when I was preparing for this talk. It was a fairly short talk (I had 20 minutes and I finished in about 15, including the one question) and it was the first one of the night talks, so I was quickly into attendee mode for the rest of the conference. There was an interesting talk on browser extension security in the night talks track, given by a student, Abhay Rana. It gave an overview of the javascript context model of Firefox and Chrome, and then went on to talk about the issue of extension writers asking for more permissions from the framework. Not exactly my primary interest (which is system software and applications security as opposed to web-based stuff), but was interesting nevertheless.

The main conference did not have a lot of things that interested me greatly, because heuristic analysis, penetration testing and fuzzing seemed like the primary focus and also the fact that there was little presented in the Free Software space, i.e. security research on Linux and/or BSD systems and software. I was even more disappointed when I found out that Chris Evans could not make it and was told that another Google engineer would give a replacement talk. Replacement talks are usually very high level, templated and not a lot of fun as a result, but I was in for a surprise. Sumit Gwalani talked about Chrome browser and OS security and for me that was the best talk of the conference. I had a very useful chat with Sumit later about some aspects of glibc and memory allocation tweaks that Chrome does.

Other than that, there were a number of hallway talks and discussions with attendees and speakers over interesting topics like reversing programs, binary patching and malware unpacking. The Bogmallo beach was probably the most beautiful Goan beach I have been to till date, with friendly people and great food. The Bogmallo beach resort is good, but overpriced a bit.

The GNU C Library project released version 2.19 of it’s library on Saturday (Friday for some), with Allan McRae as the release manager. Apart from numerous bug fixes, there are a couple improvements that would interest developers. Both improvements are related in some manner to the library documentation, which apparently is not very well known. In fact, it seems that very few people know that the official documentation for functions in the GNU C library is not the man pages, it is the GNU C Library Manual. This is not to discredit the man page project in any way of course - Michael Kerrisk does a swell job of keeping man pages in sync with glibc behaviour wherever necessary and I’d like to think that we’re cooperating to the best of our abilities. The man page project however is more general and covers a fairly broad set of components including the kernel, the C library and some tools. The glibc manual focusses specifically on functionality provided by glibc.

Now for the first big improvement. Alexandre Oliva, along with a number of reviewers did the mammoth job of adding documentation regarding multi-thread safety, async-signal safety and async-cancellation safety for functions provided by glibc. This is an invaluable resource because it tries to describe precisely what kind of guarantees the glibc implementation of various functions provides, as opposed to the guarantees documented in the various standards.

The second improvement is the introduction of Systemtap userspace probe markers for various events in malloc and some mathematical functions. The malloc subsystem is fairly complex and has some critical events that a developer may want to track when trying to profile memory allocation patterns for their programs. These probes are placed at such critical points in the malloc subsystem so that one may write up a systemtap script to profile their applications much more easily. I had written a description of the malloc internal implementation some time ago, which is still relevant and may help developers select the appropriate probes.

Some mathematical functions try to provide a guarantee of accuracy of the result to the last bit and to do so, some inputs may require multiple precision computation (I have of course, written about this in a bit more detail in the past). This fallback computation may be in multiple stages and may take anywhere between a 100 times to a 1000 times more than the normal execution speed of the function. While this fallback is needed only for a handful of inputs in the entire range, the performance impact is really high when an application does hit this path. So to help developers identify whether their performance hit is due to these multiple precision fallback paths, these paths have been marked with probe markers that can be used in systemtap scripts to profile applications. The probes have been documented in the libc manual, in the Internal Probes section.

Finally, I have managed to finish producing a significant set of benchmark inputs for the math functions I care about, so it might be a good time for folks to start trying them out and sending in results. The README in the benchtests directory should be a good starting point. The output file format is still not final - I’m toying with JSON as the final format - so expect changes there in future. The string benchmarks still need some attention, which hopefully will happen in the 2.20 time frame.

Looking forward to 2.20, Joseph Myers has already begun the work of moving the architectures in the ports directory to the regular source tree. Once this is complete, we will have no concept of secondary architectures, which is a good thing. Hopefully in future we will also get rid of the libc-ports mailing list, which I have complained about in the past as being an unnecessary separation.

On the benchmarks front, we’ll be moving to python as the language of choice for the scripts and adding features such as graphing, a better file format, scripts to compare benchmark outputs and anything else that catches my fancy during the next 6 months.

Finally, I’ve mentioned this before - The GNU C Library manual needs contributors and here’s how you can help.

While doing a code audit, I noticed that netgroups queries resulted in warnings in valgrind on my system:

==14597== Invalid read of size 1
==14597==    at 0xBB735E0: _nss_files_setnetgrent (files-netgrp.c:106)
==14597==    by 0x4F4954E: __internal_setnetgrent_reuse (getnetgrent_r.c:139)
==14597==    by 0x4F49879: setnetgrent (getnetgrent_r.c:181)
==14597==    by 0x4033EA: netgroup_keys (getent.c:493)
==14597==    by 0x402370: main (getent.c:1011)
==14597==  Address 0x51fe63f is 1 bytes before a block of size 120 alloc'd
==14597==    at 0x4C2A45D: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==14597==    by 0x4EA403D: getdelim (iogetdelim.c:66)
==14597==    by 0xBB73575: _nss_files_setnetgrent (stdio.h:117)
==14597==    by 0x4F4954E: __internal_setnetgrent_reuse (getnetgrent_r.c:139)
==14597==    by 0x4F49879: setnetgrent (getnetgrent_r.c:181)
==14597==    by 0x4033EA: netgroup_keys (getent.c:493)
==14597==    by 0x402370: main (getent.c:1011)

The code was very obviously buggy too:

         ...
         while (line[curlen - 1] == '\n' && line[curlen - 2] == '\\')
           {
             /* Yes, we have a continuation line.  */
             if (found)
             ...

So if line has just the newline character, curlen will be 1 and hence, line[curlen - 2] will be the byte before the array. This kind of thing is never caught normally because of two factors:

1. The byte before line is part of the malloc metadata
2. The byte is read and not written to, hence there's no question of corrupting data and triggering one of malloc's consistency checking mechanisms.

Hence the code never crashes and we think that everything is just fine until the byte preceding line just happens to be 0x5b, i.e. ‘\‘. That will again never happen in the normal case since it would mean that the size of the line is at least 0x5b00000000000000 on a 64-bit system. However, even if this happens, it may not do a lot of harm since all it does is resulting in concatenating an empty string to the subsequent netgroup line.

However, consider a scenario where a netgroups file was generated in a kickstart like this:

{ for i in foo bar baz; do
    echo "${i}_netgroup \\"
    for j in $(seq 1 10); do
        echo "($i, user_$j,) \\"
    done
    # Guess what! I can get away with the \ in the end if I leave a blank line!
    # How Cool is that!
    echo ""
done } > /etc/netgroup

Here, the administrator has found a ‘feature’ where leaving blank lines allows them to get away with their very simple script to generate netgroups with trailing backslashes. Now what happens if line happens to have a preceding 0x5b? The groups separated by the blank lines will get concatenated and all of a sudden you have one group with all of the entries being members of the first group!

So if an attacker manages to control this preceding byte in a program (possibly through a different heap overflow bug), (s)he could manipulate netgroup memberships and cause a breach. So we have a security vulnerability on our hands! Or do we?

Not such a big deal after all

So while it is fun to imagine scenarios to exploit code (who knows when you’ll get a hit!), this one seems like just a harmless buglet. Here’s why:

Controlling the preceding byte in such a manner is extremely difficult, since the malloc consistency checker should kick in before line is even allocated this offending chunk of memory. One may argue that malloc could be overridden and the preceding byte could be tailored to ones needs, but that immediately takes out the interesting attack scenarios, i.e. the setuid-root programs. If you’re doing authentication using a non-setuid command line program then you’ve got more serious issues anyway because one could simply override getnetgrent_r or similar user/group/netgroup browsing functions and bypass your check.

Finally, this ‘exploit’ depends on the netgroups file being wrong, which is again extremely unlikely and even if it is, that’s a problem with the configuration and not a vulnerability in the code.

In the end it is just a trivial off-by-one bug which must be fixed, so it was.

Update: I have now updated the post with a solution that actually works. There is a disclaimer though: future system updates will not update your Email and Exchange packages; you’ll have to always do it manually.

I once had a Samsung Galaxy S (GT-i9000). It was a beautiful phone that I used, rooted and installed Cyanogenmod on (right up to JellyBean) and thoroughly enjoyed. And then I decided that I wanted to fix the GPS contact on the phone using a documented soldering solution. So that’s how I got to order a Nexus 5 :D

Jokes aside, I was thrilled to get my hands on a Nexus 5 last week. My Galaxy S had served me well for many years mainly due to great quality hardware back then and I chose the Nexus 5 for the same reason - a quad core processor and 2GB RAM will surely last me a while. There was one problem though and as I found out, it was a fairly widespread problem - corporate emails that use ActiveSync would not work. The problem was documented and also fixed, but there was no fix available for the Nexus 5 yet, unless you had Cyanogenmod.

I had decided to not root my phone for at least six months and even after rooting, not install Cyanogenmod for another six months, so my chances of getting a fix were dependent on Google releasing an update. This looked like a good chance to get my hands on some android patching and building, so I decided to give it a go. To summarize, I did the following:

• Rooted my phone
• Locate the fix
• Built the new apks from the AOSP source
• Removed the Email apks
• Installed the new apks

My phone is alive and syncing emails, so here’s a more detailed description of what I did. I am not going to write about rooting the phone or setting up the android development environment. That stuff is well documented - just make sure you get the binary images from reliable sources, like the original website.

Locate the fix

Comment #174 in the bug report pointed to the Cyanogenmod patch review system which had the patch that fixed the problem. The Google engineers unfortunately were not helpful enough to make any such note. I verified in the downloaded AOSP source that these changes were in place. I just stuck to master and did not bother going for any specific branches to do a backport because (1) it’s java code, so it ought to be largely device independent, i.e. it shouldn’t break anything on my phone if it’s wrong and (2) I’d expect that it should be possible for the apps to be independently built and installed, which they were. Since the code on master and building with java 1.7 did not work, I installed the 1.6 jdk and checked out the android-4.2.2_r1 branch in AOSP. In fact, the code on 4.2.2_r1 branch will not even build with java 1.7. On the code end, the CyanogenMod change was slightly different, but the problem was in fact fixed on the 4.2.2_r1 branch with this revision:

commit d92a75c707461188e8743149476e8f49ef191b42
Author: Tony Mantler <email removed>
Date:   Fri Nov 15 12:45:53 2013 -0800

    Make sure the client certificate is always installed
    
    b/11678638
    
    Change-Id: Iafe200d14b72678324758fe08b03c8ea7bb9dc5c

So there was no need to actually patch anything.

Build the packages

Building the individual packages is very simple:

$ make showcommands Email Exchange2

Defining EXPERIMENTAL_USE_JAVA7_OPENJDK allowed me to use OpenJDK to build java programs instead of the proprietary Oracle Java. showcommands is an additional target that gives a verbose mode. Email and Exchange2 are the package names. One could get those package names by looking for the LOCAL_PACKAGE_NAME variable in the Android.mk in the package directory ($(srcdir)/packages/apps/).

Remove the old packages

Before removing, always backup. It’s easy to do this from adb, using the commands:

$ adb pull /system/app/EmailGoogle.apk backup/
$ adb pull /system/app/Exchange2Google.apl backup/

Once this is done, get into adb shell and get root:

$ adb shell
phone:/$ su

You’ll get a prompt on your phone confirming root access, which you need to allow. Now remount the /system filesystem in read-write mode using the following command:

phone:/# mount -o remount,rw /system

Once the remount succeeds, remove the package files using the commands below. Also remove the odex files since they’ll be regenerated for the new packages:

phone:/# rm /system/app/EmailGoogle.apk
phone:/# rm /system/app/EmailGoogle.odex
phone:/# rm /system/app/Exchange2Google.apk
phone:/# rm /system/app/Exchange2Google.odex

Install new packages

Installing the newly built packages is also just as simple. First, copy the packages to your sdcard:

$ adb push out/target/product/generic/system/app/Exchange2.apk /sdcard/
$ adb push out/target/product/generic/system/app/Email.apk /sdcard/

and then copy those packages to /system/app/ and give them appropriate permissions:

$ adb shell
phone:/$ su
phone:/# cp /sdcard/Email.apk /system/app/
phone:/# cp /sdcard/Exchange2.apk /system/app/
phone:/# chmod 644 /system/app/Email.apk
phone:/# chmod 644 /system/app/Exchange2.apk

Reboot your phone and let the phone optimize your app (i.e. generate the odex file) for your phone. ActiveSync should now start working on your phone!

I had to do some analysis on the nscd database file this week, for which I hacked up a quick program to dump the contents of an nscd database file. I intend to post the code on the upstream mailing list when it is complete, but for now I wrote up a description of the file layout on the glibc wiki since there isn’t any documentation around it.