It all started when I came into discussions with the first company (owned by friends) that I have mentioned in some previous posts of mine. These gentlemen mentioned fax as a possible commercial use of my adapter. And it’s true: even in the case of an Asterisk installation with IP phones, one always needs one (usually one is enough) FXS port to connect a good, old analog fax machine.
So the question came to me: can Open USB FXS do fax? From this — misleadingly simple — question, a whole new adventure in the deep seas of debugging was born. I still do not quite understand in depth all the issues that arose but, since my stories usually have a happy end, I managed to resolve these issues. Maybe some of the readers will understand more than I do, and they are welcome to submit their comments. Good, let’s start then.
A primer on fax in VoIP systems. Usually, this is a no-no-no-dont-do-it thing. Fax machines are designed to work over the TDM network, under the assumption that there are no such things as packet loss or jitter. In contrast, VoIP systems operate under the exact opposite assumptions: packets may be lost, delayed or even get duplicated by the network. For these very reasons, it’s permissible to pack audio samples into jitter buffers at the receiving end and play all sorts of tricks to make the audio result “better” in case of losses etc. One common such trick is, for example, to repeat the last audio sample in case the input buffer is empty. This will “sound better” than sending a sample of silence. Thus, implementing tricks like this one, drivers for devices like Open USB FXS allow slack in various degrees. This should not be the case for fax, though. When dealing with fax, a single lost packet can have a detrimental effect in the reproduced image, or even cause the entire transmission to fail.
For this reason, ITU-T has come with the T.38 standard. According to that, two terminal VoIP systems that wish to exchange facsimile messages terminate the fax encoding realm locally by decoding the signals that the sending machine transmits, and then exchange the decoded contents as IP packets, using UDPTL. How about Asterisk and T.38? Currently, there is no “native” support of T.38 in Asterisk, but various commercial add-ons and public patches (such as this one) claim to add the necessary functionality.
What this situation leaves us with is the local (pseudo-TDM, in Asterisk jargon) packet handling by the driver. In other words, if T.38 handles (successfully or not, that’s another discussion) the network-facing part of the problem, the device-facing part — the device driver — should handle audio packet transmission without a single loss. Did Open USB FXS fulfill this requirement? An idea, suggested by a friend, was to try local fax transmission between two FXS ports on the same machine. I was confident enough that Open USB FXS would pass the test easily.
I was wrong. The two fax machines that I tried were unable to pass the initial negotiation. When I eavesdropped the phone line, I discovered why. The tones that the answering fax machine was sending were arriving badly clipped and distorted. But why was that? Voice audio was delivered fine; why would the fax tone be clipped?
The answer was simple: high frequency. A quick test revealed that high frequencies were somehow “filtered” by Asterisk when I used Open USB FXS. It did not take me long to understand the next culprit in the chain, multi-sample packing per URB. My driver tries to be smart and make best use of available system resources, in order to perform acceptably even on low-power CPUs (I am emulating such an environment using VMWare). In order to do that, it packs multiple 1-ms audio samples into a single URB [see why this is more efficient by consulting the discussion at the end of this post]. When the URB completes, the driver passes all samples together to the dahdi core (and from there on to Asterisk).
Apparently dahdi did not like that. I took me quite some time to test all possible combinations of driver parameters, until I finally noticed that the clipping effect was lesser when I was loading the driver with only two samples per URB (using the “rpacksperurb” and “wpacksperurb” parameters). Still, the quality was not quite acceptable for fax transmission, but at least one could make out the familiar sound of an answering fax machine sending various tones. This encouraged me to look deeper and test further.
The number two (two samples per URB) seems to be related to the buffer depth of dahdi, which is exactly two samples. So it’s no wonder that submitting more samples at once to dahdi screws things up, because it results in newer samples overwriting older ones before the latter get the chance to be read off the buffer. Interestingly, there is no user-side way to change this, because although there is an ioctl() to change the number of buffers in a channel, a subsequent close() resets this to the default number 2. Even more interestingly, changing 2 to a higher number (8) in the dahdi source and recompiling dahdi did not make the high-frequency fax tones sounding better. I declare my ignorance as to why this happens, but changing the dahdi source did not feel like the way to go, so I went back into trying to fix my driver instead of fiddling with the dahdi code.
I tried with one sample per URB, but the results were still not what I was hoping for. Sometimes I was getting good results, sometimes not, and I could not understand why. It was only then that I finally thought about removing the dahdi_dummy timing module. Dahdi_dummy employs the high-resolution timer available in the x86 platforms to “tick” the dahdi core once per millisecond. However, this “ticking” is not synchronized in any way to the pace of interrupts arriving from USB. I am not sure why, but this caused still bad quality audio in high frequencies, even with a single sample per URB.
To make things worse, the default initialization script /etc/init.d/dahdi checks for hardware dahdi channels, and if it finds none, it loads automatically dahdi_dummy. I have the bad habit of plugging my USB dongles after the system boots and watch them initialize through dmesg messages), so dahdi_dummy was loaded by default. This caused me much confusion, because I got some good results which were worse again after a reboot, and it took me a lot of bumping my head on the wall to find out why: it was dahdi_dummy, which was being reloaded after the reboot, and I was forgetting to remove it.
After all that, I tried again the test with the two local fax machines. This time everything went fine. I sent over an image (an ad of a Bob Dylan concert, you can make out the greek letters on it) from a Samsung FS5100. Here it is:
Here is the transmission report, showing what the sending fax machine sent to the other side (the handwriting is mine). Note in particular the noisy parts (e.g., the dots inside the white letters “Bob”), which are already present in the scanned image:
Finally, here is what the receiving part got — a perfect reproduction of the transmitted image:
From all that, I guess Open USB FXS deserves the title of this post, F(a)XS(uccess)! However, here are some final notes. In contrast with a PCI(e) driver, where a few outb/outw instructions suffice to handle an interrupt, submitting an isochronous URB is a very expensive operation. It takes a considerable number of CPU instructions, and this is mainly because the driver must make sure that the submitted URB does not take up too much out of the available bandwidth on the USB bus. Especially when plugging more than one Open USB FXS devices into the same computer, this could result in bad performance, loss of USB timeslots etc. On my VMWare testing platform, this is apparent, in that the kernel reacts slower than required and misses USB timeslots, resulting in audible “clicks”. I have looked extensively into the kernel source for UHCI, trying to find a way to get an interrupt per USB frame (not per URB completion), but it seems that there is no such thing. In theory, I could use in my driver the HR-based “ticking” of dahdi_dummy and test the first submitted URB for per-packet completion, even with more than one samples per URB. However, this seemed to me very complicated, so I let it go for the time being. Other drivers (Astribank xpp for example) seem to do something like that. Maybe that’s the way to go in the future.
I am now getting back into the prototypes hell (which BTW have some issues, and this is why I am holding back the whole release phase), but there are lots of things in that hell to blog about, and this would deserve a different post. Soon to be, I hope, along with the announcement of public availability!