Archive for March, 2013

A trip in the dry lands of maintenance and compatibility

March 16, 2013

Super hackers and the VM potential

After an admittedly-long-time-no-see period, I decided to add this new post about compatibility with various Linux flavors. The reason is that, throughout all this period, while I did not make any progress on Open USB FXS (and FXO either), the globe kept spinning, and new versions of things kept popping up. New versions of the Linux kernel, new versions of Dahdi, and of course new versions of the Linux distributions that I had been writing about. All this otherwise welcome wave of progress was driving into obsolescence my older instructions for building and installing the Open USB FXS Dahdi driver.

I have to admit that this project tends to require a sort of “all-in-one” hacker to understand and build. One must know about electronics, have experience with SMD technology, get involved with USB intricacies, understand VoIP issues, and — worst of all — be knowledgeable in the Linux kernel and device drivers — not to mention a requirement for moderate experience with the Unix/Linux shell, editors, file system structure and commands. Although some of the fans that this project has attracted through its lifetime fitted this profile, quite a few did not. Both sides have been represented: electronics fans who did not know much about Linux; and Linux gurus who did not wish to get their hands dirty with soldering. People who did not wish to get involved with chasing and soldering louse-sized components have (or used to have, my stash is now almost extinct) an alternative: try to get their hands on a ready-made board. This covered indeed one end of the spectrum. However, the other end was not so easy to satisfy: people with experience (or a great love for, or both) in electronics, who felt at ease with soldering irons, SMD, flux, magnifying lenses and multi-meters, who however had little or no experience in Linux were facing problems. What distribution should they choose? What to put in dkms.conf? Why did the compilation result in error on distro X.13.1.20-2? And so on.

I wish I could provide a way to help these users. Apparently, I could not — I can not — ship a ready-made PC or other embedded system with Linux and my drivers installed, and even if I could, the cost would not be justified. One of my thoughts had been to provide a ready system, built as a VM, for people to download and experiment with. I do not know if people find this a good idea, however there are some shortcomings to it. The main problem is that, as documented through various posts in this blog, VMs do not perform well enough: an isochronous driver on a VM tends to miss USB slots and this results in bad audio and other issues. Thus the usefulness of a preconfigured VM would be limited to just proof-of-concept or testing environment, where people would be able to test their self-made DIY Open USB FXS board. The actual value of the VM concept let aside, I was — and still am — feeling terrified by the prospect of maintaining separate Linux distros as VMs, as per user requests. What if I provide, say, a Debian VM, and then someone comes in saying “please, please, do also provide an Ubuntu VM”? And what if I do provide a Ubuntu VM for version X.Y.V-12 of Ubuntu and then version X.Y.V.-13 pops up, so I need to align?

For the exact same reasons, in my previous posts I had refrained from giving out instructions alleged to fully work on a given version of distro X. What if I provided good and valid instructions for a snapshot of distro X, and distro X then fired out an update (say, a kernel update) that would brake it all? [And mind you, this is no science fiction; I found this to be exactly the case with Ubuntu 12.0.4LTS. Not with my driver. With their own provided ones. Read on for the details.] Apparently this would not work. Linux users are — and have always been — required to a certain degree to take control of their spaceship and perform themselves some system housekeeping and maintenance, (including some googling and guesswork) to keep up with issues like these — especially when it comes to distributions that consider spitting out untested updates on a daily basis is a good maintenance practice.

All the above fail to include some nasty — though I hope imaginary — support requests that come to my mind, like “I need to have VMWare tools installed”, or “my VMWare version does not support your VM”, or “your VM image does not see my USB devices”, or whatever else. No, no, please, take this cup of suffering away from me…

Nevertheless, if there are people who find all this crazy VM business a good idea (and promise not to ask me to keep up to date with the crazy rate at which Ubuntu and others fire updates of their systems), then I might go into that. Only for demo and proof of concept purposes, right? And I will not provide anything more than a couple of demo VMs and your favorite distro or version thereof may not be among them. And they may not work on your VMWare player or whatever. And I will just provide them as-is and they may be outdated. And if you bring them up and let them install updates from their distro repository and this breaks everything I shall not be the one to blame. And I will not provide support or updates for them. Agree? If yes, please post a comment or mail me and let me see what I can do about VMs.

Still believing that the best idea would be not to provide VMs but just good instructions to help people compile and install the drivers on their own barebone Linux system — no VMs –, I went on testing the status of my driver with several systems and kernel versions.

Debian 6.0.6 (“squeeze”)

Debian used to be very straightforward in the past, and proved to still be so. My main concern, which lasted only about ten minutes or so, was to install all the required packages. I re-synthesizing the packages I selected by hand by looking at dpkg’s log file, so I may forget something, but essentially you need the following: the toolset to build kernel drivers, the dahdi binaries and the dahdi source, subversion (to checkout my oufxs driver) and libusb-dev (because some programs in my driver depend on it). So, you need to install gcc (if it isn’t there), dahdi, dahdi-linux and dahdi-source, kernel-package, linux-headers (and maybe linux-source, I am not sure if this is required anywhere), subversion and libusb-dev. Just use apt-get install package where package is one of the above packages. Then,

root# cd /usr/src
root# mkdir dahdi-src
root# cd dahdi-src
root# tar jxf ../dahdi.tar.bz2
root# cd modules/dahdi/drivers/dahdi
root# svn checkout http://openusbfxs.googlecode.com/svn/trunk/LinuxDahdiDriver/oufxs
root# cd ../..
root# make SUBDIRS_EXTRA+=oufxs
root# make SUBDIRS_EXTRA+=oufxs install
root# modeprobe oufxs
root# lsmod | fgrep oufxs

, and there you have it. (Note that the long “svn checkout” line above is wrapped, so do not copy-paste blindly from your browser into the command shell). The last ‘lsmod’ line is not strictly required, it is just for you to see that the module has been insmod’ed correctly. You should do the rest of the configuration and testing as described at my other blog posts (i.e., for installing Asterisk, setting up startup scripts and default module options, testing with fxstest, etc.).

Ubuntu Workstation 10.0.4LTS

Ubuntu 10.0.4 is also straightforward. In terms of dependencies, things are very alike with Debian squeeze. Apart from the obvious packages required (gcc etc., which I think get installed by default), one needs dahdi-source, dahdi-linux and dahdi-dkms (these depend on each other, so you may get away with just one of them), subversion and libusb-dev. When installing dahdi-dkms, dpkg (the Debian-originating package installer) invokes dkms, so your system has now already dahdi installed through dkms. The next steps are then (a) tell dkms to remove the Ubuntu-supplied dahdi driver tree from the dkms tree (and from the kernel), (b) download the oufxs driver, (c) fix dkms.conf and (d) rebuild dahdi the dkms way, this time with oufxs in. So, here we go:

root# dkms remove -m dahdi -v 2.2.1+dfsg-1ubuntu2 --all
root# cd /usr/src/dahdi-2.2.1+dfsg-1ubuntu2/
root# vi dkms.conf

Now, you have to go to line 5 (or so), starting with MAKE[0]=, and make that infamous and dreaded change to SUBDIRS_EXTRA (in the past, I have provided some not-so-accurate instructions, thus people kept on asking how to make this change). The line must now read (note again line wrapping – this should be one line on your editor):

MAKE[0]="make modules KERNVER=$kernelver MODULES_EXTRA='opvxa12
00 wcopenpci dahdi_echocan_oslec' SUBDIRS_EXTRA='../staging/ech
o zaphfc/ oufxs' ;make;make firmware-loaders;echo : > drivers/d
ahdi/vpmadt032_loader/.vpmadt032_x86_32.o.cmd;echo : > drivers/
dahdi/vpmadt032_loader/.vpmadt032_x86_64.o.cmd;make"

Beware to include oufxs inside the quotes, like this: SUBDIR_EXTRA=’../staging/echo zaphfc/ oufxs’. Then, go to the end of the file, and add the following three lines before the “AUTOINSTALL=yes” line:

BUILT_MODULE_NAME[35]="oufxs"
BUILT_MODULE_LOCATION[35]="drivers/dahdi/oufxs"
DEST_MODULE_LOCATION[35]="/kernel/drivers/telephony/dahdi/oufxs"

Note that the number 35 above is the number next to the last one used by the ubuntu-provided dahdi source. In my case this was 34, so I picked 35. If Ubuntu update their dahdi-dkms package to add or remove drivers, this number may change and you will have to adapt accordingly. Then, you have to perform the following steps:

root# cd /usr/src/dahdi-2.2.1+dfsg-1ubuntu2/drivers/dahdi/
root# svn checkout http://openusbfxs.googlecode.com/svn/tru
nk/LinuxDahdiDriver/oufxs oufxs
root# dkms add -m dahdi -v 2.2.1+dfsg-1ubuntu2
root# dkms build -m dahdi -v 2.2.1+dfsg-1ubuntu2
root# dkms install -m dahdi -v 2.2.1+dfsg-1ubuntu2
root# modprobe oufxs
root# lsmod | fgrep oufxs

Notice again the line wrapping on the “svn checkout” command above — this should be one line.

Ubuntu 12.0.4LTS: the story

OK, let us now come to the latest-and-greatest version of Ubuntu. Lots and lots have changed in this version. Starting from the kernel version and extending to the screen resolution, the incompatibility of their own packages with their own-supplied udpated kernel version, and not forgetting that synaptic is not anymore installed by default, yielding its place to the “Ubuntu Software Center”, this version of Ubuntu is guaranteed to cause more headaches with Open USB FXS. So, here is how my story goes with 12.0.4. I hope you will find useful information in the course.

To begin with, I do not have a host of spare PCs to test on three different OS’s — to tell the truth, I have not even one spare PC; I do all my work on VMs. This time, I started by installing Ubuntu 12.0.4LTS on a new VM, but I had to interrupt and thus I suspended the VM in the course of installation. This is guaranteed to work on other OS’s. but on this one, resume gave me the following insightful message:

To revert to this snapshot, you must change the host pixel format to match that of the guest. The host’s current settings are: depth 24, bits per pixel: 32. The guest’s current settings are: depth 24, bits per pixel: 32. Error encountered while trying to restore the virtual machine state from file “…\Ubuntu12.0.4LTS.vmss”.

No other action than power-reset of the VM was possible at this stage, and of course, since I had paused the VM halfway through the initial installation, this gave me a black screen. Starting from scratch, I got 12.0.4 installed propperly without pausing the VM in the process. But then, at some later point, I had to pause and resume gave me the same error. A little googling got me to this post, which described accurately my problem, and was pointing to the right (partial) fix. I am saying “partial”, because it seems that the user must be logged on for this fix to work, but nevertheless, this was a great boost in productivity, comparing to having to reboot each time I needed to pause VMWare.

So I let the update manager do its own work. Funny enough, the first few times it tried, the update manager complained about being unable to verify the signatures of Ubuntu’s own packages. A bit of googling and reading on this gave the answer: just let the damn thing its time, and it will work. Indeed, it seems that 12.0.4LTS and its update manager need to fish the signature keys somewhere, and this somewhere takes time to respond or isn’t always up or whatever — I really did not care to find out the exact truth. I just noted the lousy user experience of bringing up a system that fails to update itself complaining that it does not know the signatures of its own vendor’s packages, but I said, anyway, what the heck. Letting it live for ten minutes or so got me the keys out of the blue, and the update manager did its thing and updated the system, kernel included. Wow, now this system runs 3.5.0-25! Yeah right… just read on to the next paragraph.

Next surprise was synaptic. It was gone. OK, I admit that I could have done my work using apt-get, however I am a fan of synaptic because it helps searching out through packages. Strangely enough, I was able to find dahdi in the “Ubuntu Software Center”, and I selected it for download, and… guess what: dahdi failed to install. But this was crazy! A Ubuntu-provided package fails to work on a Ubuntu-provided and Ubuntu-updated system! As always, a deeper look helped spot the problem. The dahdi version 2.5 that Ubuntu includes fails to compile with the kernel version 3.5 that Ubuntu guides you into updating to! The culprit is this little change after kernel version 3.3, which I read has broken quite a few third-party kernel drivers. Fortunately, googling found me an easy fix for this, so all I had to do was to un-install the Ubuntu-provided dahdi dkms package, fix it myself, and re-install it:

root# less /var/lib/dkms/dahdi/2.5.0.1+dfsg-1ubuntu2/build/make.log

got me the culprit file, wcb4xxp/base.c, which I had to fix according to the above fix. The steps:

root# dkms remove dahdi/2.5.0.1+dfsg-1ubuntu2 --all
root# cd /usr/src/dahdi/2.5.0.1+dfsg-1ubuntu2/drivers/dahdi
root# vi wcb4xxp/base.c      # apply the fix circa line 39
root# dkms add dahdi/2.5.0.1+dfsg-1ubuntu2
root# dkms build dahdi/2.5.0.1+dfsg-1ubuntu2

All this, to get me the distro-provided package installed on the distro-provided system. But it was done. Now what? From some users who had tried compiling and running my oufxs driver on newer kernels, I had reports that some tweaking had to be made in the power management part of my driver. To fix this, I had better have the kernel source and/or the kernel man pages around. Looking up for the kernel source 3.5 in the “Ubuntu Software Center” gave me nothing. Ooops… The distro-provided package repository does not give me the source for the distro-provided kernel. How come?

NOTE: the following paragraphs are a tale of my own steps. You do not need to repeat these steps.

I then decided get the familiar synaptic back, mainly to get rid of all the annoying commercial software and magazines that the “Ubuntu Software Center” presented me with when searching. This was an easy step, so I got synaptic installed. Searching with synaptic confirmed that the Ubuntu 12.0.4LTS source repository does not have the source for kernel 3.5.0-25. Great! So I had to add manually the respository of quantal (the next version) in the end of /etc/apt/sources.list:

deb http://ubuntu.mirror.cambrium.nl/ubuntu quantal-updates main

, then do an apt-get update to let apt know about the newly-added repository, then search with synaptic the kernel source for 3.5 and add it, and then (to be on the safe side) to remove the quantal repository form /etc/apt/sources.list and to re-do an apt-get update. It was getting funny. Anyway, I did all that and got me the kernel source for 3.5.0-25 – besides, I only needed to study the source, not compile it.

I then tested plugging in my Open USB FXS board. As reported, the driver initialization failed because usb_autopm_get_interface was returning -EACCESS. I checked again with the source and usb_skeleton.c, and it turned out that during all this time I had been using the usb_autopm_XXX functions incorrectly. Normally, I should use these when a process is opening the device. Instead, and in order to disable autosuspend altogether (which was my purpose after all) I needed to provide a handler function and place it into the .suspend pointer of my driver structure, making sure that this function would return zero.

This is where I stand now. During the next days, I will update this post as I am fixing my driver and finally I will upload a new version on the project’s Googlecode repository. Stay tuned, if you are interested.

Update, April 16 2013

The code has now been adapted to work with Dahdi 2.5 and the 3.5 kernel that Ubuntu runs. No substantial changes were required for the Dahdi adaptation, just a few #if’s in the code to account for minor version 5. Before I set these, I payed a couple of kernel panics and freezes, though, because the  precompiler tests were checking for minor version == 4 and not for >= 4. Lesson taught, I changed these ==’s to >= and all went fine. I had to remove usb_autopm_XXX functions altogether. However, this brings in an interesting question.

How would you like Open USB FXS to do power management?

As I said above, in order to get my driver to work, I had to remove usb_autopm functions. I was using them wrongly anyway, so no much harm was done. Supposedly, I was invoking usb_autopm_get_interface at driver initialization time to prevent the kernel from putting my dongle board in low-power mode. However, my driver was not providing to the kernel hooks for that (like .suspend, for instance), so I guess that auto-suspend was anyway disabled. Second, the autopm_get function is supposed to be invoked at file open time, whereas I was invoking it at driver initialization. The function must be called in “process context” and not in “interrupt context”, however I think I was bypassing that, because all the driver initialization was being done in a work queue (that is, in a separate kernel thread on its own), and I think this is considered “process context”. I have no idea why the kernel rejected the call with -EACCESS, but in any case that call was not doing much anyway.

However, this brought in an interesting question. How should Open USB FXS behave if it it had been engineered for supend/resume functionality? In principle, it should like go in “low-power” mode, turning off the converter and probably even shutting down all operations of the Si3210. The PIC’s USB functionality “knows” about suspend and resume, and the PIC is able to survive with exremely low currents. So, that would be a “USB-compliant” mode of suspending your Open USB FXS board. Would that be interesting, though?

My guess for an answer is no.

One could imagine another, more interesting mode of suspending a telephony device. How would you like, for example, your computer being put to sleep, and waking up when you take your phone off-hook? Or, in the case of an FXO device (assuming that I ever make it to get mine working), when the FXO interface indicates a ring from the Telco line? For me, that would mean lots of power savings. Is it doable? Certainly, for an FXS interface, it makes little sense: if we power down the converter, the device will no longer be able to track an off-hook event. However, an FXO device might be easier to deal with.

For the time, I do not intend to do any work in the area of power management. Nevertheless, I am tossing the ideas, just in case someone reads this and wants to contribute (ideas, experience, views or whatever).