Archive for May, 2010

Nothing is easy

May 28, 2010

Frequent visitors of this blog might have been wondering what the heck I have been doing all this time. Well, the answer is “I ‘ve been trying to get the prototypes business going”. This post is about all those messy details of this effort that would otherwise go unnoticed. Initially, I did not really think that all this crap was worth posting. After giving it some thought, however,  it struck me that all stumbling upon the details that follow was very similar to every other debugging session described in other posts of this blog. Here are the details, then.

To begin with, I am not a company and I do not own a company. According to the Greek Revenue/Tax laws, buying and selling stuff is “commerce” (well, in a sense this may really be so) and this is an activity that individual taxpayers are not allowed to have. Instead, individuals must form a individual-owned company, which is then required to have its own VAT number and do its own accountant bookkeeping. In real life terms, this meant that, in order to sell the prototypes, I would have to spend some weeks in various public sector services (Greece is notorious for its bureaucracy levels) to start up this company, plus a non-negligible amount of money for just being allowed to do business (I don’t know if you read the papers, but Greece is also — and lately it is becoming more and more — notorious for taxing and charging with dire cruelty everything that breathes, moves, eats or speaks). I would also need to pay an accountant to get me through the maze of legal procedures concerning financial activities, bookkeeping, logistics, etc. OK, if one is planning to open a shop or a production line, maybe all this is worth the trouble. For just selling a bunch of prototypes at bare cost price without profit, it does not really make any sense.

So then, it was clear that I had to act differently. As I already said, I came in terms with a company owned by some friends, who would undertake all the logistics, accounting, and bookkeeping, while I was to coordinate the thing. That agreement having been made, I just went on and compiled my first basket from Mouser, and then went back to my friends and said to them: “OK, we are now ready to begin, please go to this URL, use that username and password to log-in, and then a cart is waiting for you; all you have to do is click the checkout button — and pay for the order, of course”. Ideally, this would take less than one working day of waiting and less than ten minutes of actual work, but it turned out that the world is not such an ideal place… Day after day, I was trying to reach out the people I had come in terms with, ask why they had not yet checked out the basket, and kindly urge them to do so. The answer I kept getting back was, something like “Sorry, I was terribly busy today, I ‘ll do it first thing in the morning tomorrow”. And the thing went on like this.

All this delay was making me really anxious. As the hardware-experienced readers will know, electronics stock houses do not keep materials in stock forever. These days, only Mouser seemed to have the Silabs chips in-stock, and if they went out of stock, the lead time to re-order would be in the order of six to twelve weeks. OK, Mouser had a good number stocked, however nothing precluded a potential buyer from ordering the whole stock at once. The chances of that happening were very very low, but the risk of delaying the prototypes business for months was one that I wasn’t willing to take. So, every day that went off without my friends checking out that damn Mouser basket was a source of frustration for me.

After almost two weeks of anxious waiting, unanswered calls, SMS messages and mails, I decided that this was not going to work. I loved — and still do love — my friends, but if it took them weeks to click on a checkout URL, then it would take us ages to just gather all required materials — not to speak about what would happen with assembling the boards. Fortunately, I had a backup scenario already in place: a second company that I had come in similar terms with, who proved to be much more available. It took just a phone call to agree on a meeting date in order to click the checkout URLs together. That should do it, and that should be all.

Nope, it wasn’t all. To begin with, I had to create a bunch of new web accounts for the second company, something I had already done once for the first one. Sure, that was not that hard — at least not in principle, because at the Elektor PCB service/Eurocircuits site, where I decided to re-order my prototype PCBs, another unpleasant surprise waited for me. I created the account, I uploaded the latest Gerber PCB files, I specified the quantities, I calculated the price and then — where is the “Submit” button? Believe me or not, there was no “Submit” button! I know it ought to be there, because I had used it before. Initially I tried the Elektor site, and when I saw the bug, I tried the Eurocircuits site (the latter is an OEM of the first, and what is worse, if you try to work on both sites at once, they sometimes get confused and redirect you to one another — but this is a different story). No “Submit” button there, either.

A quick phone call to Eurocircuits did not buy me a lot. They said this was a problem they had seen before but they did not know what was causing it, they asked me to submit a screenshot and let them know my user id in order to try reproducing the problem. I sent them that, however I did not receive an immediate feedback (such as a mail saying, e.g., “we verified that this bug exists and are working to fix it”). Then, after spending some good time on their web site, I accidentally browsed to the accounts management page. There, I suddenly noticed that the company account I had just created had a missing field. This field was called “Initials”, and was not present at all in the registration page, so it would have been impossible to fill it in at first. Nevertheless, on the accounts management page, this very field was marked “required” with a red star. As soon as I put some crap text in it (I didn’t really know what “Initials” would stand for to make it a required field), submitted the change and went back to the orders page, the “Submit” button magically appeared. Shoot…

Nope, this was not all. Together with my friend at the second company, we went through the submission with Mouser, he gave his credit card information and all went fine. Not for long, though: some hours later I received a mail from Mouser’s credit department, saying that the credit card had failed authorization (jeez, why did not that show up when we were checking out our order?). However, my friend was swearing that he had payed for this credit card this very morning. So, what was wrong then? The truth did not shine until a day later: the company had a special arrangement with its bank, with two company credit cards that were fed from the same company bank account. However these two were still behaving as two different credit cards. When money was deposited to the bank account, that did not act as a common pool for both cards; instead, using some algorithm — unkown, even to the bank’s representatives to which my friend spoke over the phone — money was put into one of the credit cards (e.g., the one with the shortest remaining time limit, or the one with the highest debt — the bank’s representatives were not able to tell). Obviously then, my friend had used the wrong card, since money had gone to the other one. [If you find that doing business that way is fascinating, I can give you the name of the bank]. Running the order again with the new card fixed the problem, and now most of the required materials were on their way. Pheeww!…

Not all materials are from Mouser. Some I have ordered from Farnell, and some others I was able to find in retail stores cheaper than I would find them in any components stock-house. So I am still waiting for more materials to ship or to arrive. I have to say that this whole optimization work is very cumbersome. It is also very frustrating to find out right after you have ordered 200 pieces of component XYZ costing something like 50 EUR/USD, you could just as well have ordered the same component from retail or from another stockhouse at half the price. Sometimes even these numbers are misleading: Digi-key orders are subject to import taxes, so one might pay an undetermined amount on the top of the list price of each material.

The bottom line is that it requires an expert in the electronics market to make this whole business run as cheap as possible. I don’t know how much of an expert I am, but I tried to optimize as much as it made sense the cost of my BOM. I have decided to offer 50 prototypes (assembled and DIY kits included), so the economies of scale I could achieve were elementary. With the optimizations I was able to make, the BOM, including PCBs and all materials, amounted up to something between 1,200 and 1,400 EUR. There are some cases where VAT would be added, so this would be higher (if you allow me to become a bit sarcastic, Greece –striving, as maintained by its political leaders, to make its economy more competitive — has increased the VAT coefficients from 19% to 21% and then again from 21% to 23%, and this means that imported materials from other countries might cost 23% higher than if I set up this whole prototyping business in China or India — hey, Mr. Government, you’re doing a great job in saving my country! Keep it up like this!). Anyway, this means that a bare DIY prototype (no assembly, no testing, no guarantee it is going to work) will cost something in the order of 35.00 EUR plus P&P.

Besides optimizations based on large quantities and cheapest material selection, there are quite a few optimizations that I plan making on the board itself. For example, the double SMD DIP switch is not cheap at all. In a production version of the board, this would not need to be present, since it is easy to tell the firmware to jump to Flash-programming mode by just a simple command over USB. If need be, an SMD open contact can act as a switch by short-ing its contacts when the board boots. And there are many other optimizations, only I feel this is not exactly the time to start this discussion. Maybe in the next post.

I hope you enjoyed this post. To get to its title, what else could that refer to if  not to a song’s lyrics? It’s an early one from Jethro Tull, in which then-young-and-optimist Ian Anderson gives some piece of advice:

Nothing is easy. Though time gets you worrying
my friend, it’s o.k.
Just take your life easy and stop all that hurrying,
be happy my way.

But maybe I ‘m just not that type of guy — or am I? Maybe neither Ian Anderson is that type of guy, or else he wouldn’t have written all that wonderful music of his (music is like any other business: it requires endless hours of study, trials, rehearsals and failures before even one inspired and well-performed musical phrase can come out).

Nothing is easy, then, and prototypes are not an exception. But now materials are on their way, and my motto is that getting something going is more than half of the work. In my next post I will most probably announce some details on how to order boards and kits. In the meantime, we can all rely on young Ian’s optimism. Maybe he knows better than we do.


Good news for the prototypes

May 7, 2010

I have now finished with the code for the hook debouncing and for invoking the echo canceller. The latter is still largely untested, but I might be able to set up a test maybe even today. One could say that the code is more or less ready now. The only thing that has not yet been ported from the old driver is the code that counts USB packet-level statistics (missing sequence numbers, incorrectly received USB frames and the like). At this point in time though, I feel that these are not really urgent, since I have kind of resolved the issues for which these statistics were needed. So, I am going to update the SVN tree soon with the latest version of the driver (and will place an update to this post as soon as I do that).

One thing that I need to do is to write some compiling instructions on how to best compile the module on various popular Linux distros. When trying to compile the driver, one’s mileage may vary depending on the distro/kernel/Kbuild environment. To give an example: because of a bug (mentioned in an older post of mine), in Debian lenny one has to either issue a “make EXTRADIRS=oufxs” at the top-level dahdi directory, or else the Module.symvers file has to be copied manually for an isolated “make” to work inside the oufxs directory. On ubuntu, “make” inside the oufxs directory fails altogether with a syntax error (?) message; plus, in order to install the driver via the dkms facility, one needs to fiddle with the dkms.conf file found in the top-level dahdi directory. I have not the slightest idea what the situation is on other distros like Red-Hat ones.

I have managed to bring the failed board mentioned in item #6 of my previous post back to life! The problem was that the 3201 had not been soldered to the thermal pad and thus had burnt after some time of continuous operation. I suspected the 3201 because of the following symptoms: the board would not make the phone ring at all; nothing was heard on the earphone when the card switched to forward active mode (a click is normally heard); and, the DC-DC converter was producing correctly 65V on powerup, but the voltage fell to 9V when the card switched to forward active mode. All these indications suggested an issue beyond the 3210, at the output stage. Replacing the 3201 with a new one — which I tried to make sure is now tightly soldered to its thermal pad — resolved the problem and the card is now alive and kicking.

My third and last dongle-version-2 prototype card remains out-of-order. The indications there suggest a problem in the 3210 or in the PCM bus between that and the PIC. I need to spend some time using an oscilloscope in order to debug this further, and I don’t know if it is really worth the trouble at this stage of the project. I might do it later on.

I have to note down that I owe to try out the pull-down resistor suggested by Gabor in this comment. The truth is that I am having some noise, which I have been overlooking all this time. Probably, as the fix by Gabor suggests, the cause is that the 3210’s PCM bus driver, when outputting a logical zero on the DTX line, does not offer that a good GND-level surge for the noise caught by the transmission line between the 3210 and the PIC. Thus, unless the fix with the pull-down resistor is applied, noise can possibly make it into the audio path. Another issue is the noise induced into the analog audio path, mainly from the DC-DC converter. Fortunately, the 3210 contains a squelch filter for that, however the filter takes some time to adapt after changes in the linefeed mode (direct register 64), during which a high-frequency hum is audible. Probably the solution there is to find out which indirect register(s) are related with the squelch, save their values and re-apply that after every change in DR 64’s value. Anyway, just in case I am to adopt Gabor’s fix, I have checked the PCB design and it is easy to add the pull-down, near the “via” for the DTX signal, with no major changes.

Another piece of good news is that I cannot seem to reproduce the DTMF dialing intermittent failures on a native Linux. The issue still remains on the VMWare-hosted Asterisk, which by now is well known to starve from CPU and I/O resources. Probably the failures are due to a CPU-starved Asterisk missing samples and thus being unable to decode correctly DTMF. This is a great relief, in that DTMF recognition intermittency was a scary, scary bug, at least to me.

Perhaps the most important piece of news in this post is that I have now got on in an agreement with a local company owned by some good old friends, in order to set forth the production of a limited quantity of prototypes (in both an assembled board and a DIY kit flavor). This involves a number of steps, too, and I am going to spend some time discussing these steps.

A quick inspection and review of my board by some hardware assembly specialists revealed to me that there is substantial room for improvement in the choice and placement of components in order to ease the mechanized assembly and thus reduce the cost of the board. To name one, I had better change all components into SMD ones (my board still has a number of through-hole components: two electrolytic capacitors, a tantalum capacitor, a crystal, and the USB and RJ11 plugs). In PCB assembly production lines, placement of through-hole components slows down the process and costs more.

Another area of improvement is that, because the board has components on both sides, it normally requires a second baking round in a reflow oven, and bottom-side components must be glued to the PCB, or else they will fall off the board during the second bake phase. This raises the cost of assembly. However, if components are all oriented alike, the bottom soldering can be done in a solder bath, which is much cheaper than baking in a reflow oven. So, if I am to go into mass production, I had better redesign the bottom side of the board to orient all components west-to-east (the PIC cannot change orientation, so it will make the rule for the others).

Finally, there are lots of components, mainly on the top side, which are too close to one another. Mechanized assembly tools like pick-and-place machines tend to complain or even fail in such cases. Some examples where I should allow components more headroom between them can be found around L1 (mainly the power transistor and the crystal).

The good news are that this redesign step, otherwise time-consuming, will probably be unnecessary for the production of prototypes. Because of the limited number of prototypes, we may go for a manual assembly production, without this incurring any substantial higher cost (the cost for the setup of a mechanized assembly production line is high anyway). The only possibly required redesign of my PCB at this stage involves the DTX pull-down resistor mentioned earlier.

This means that, during the next few weeks, I might be able to give out some more-or-less concrete dates for the availability of assembled prototypes. However, please bear in mind that things are now getting a bit more complicated, since more people and at least two companies are involved. So, please be patient if planned dates shift a bit. For the time, the only planned date is next week, when I think I ‘ll be able to give out some first timeplan. In any case, the important piece of news is that there is now sort of a commitment to produce and give out prototypes at a nominal price. The dates will follow.

I will make sure to include a shot of an assembled v2-dongle in this post, just to make it livelier. Other updates may follow as I get done with the echo tests and as the prototype production is progressing.

Update, May 15: The driver code with the hook debouncing and the invocation of the echo canceller has now been uploaded to the project’s Google code page. Also, I have patched a board of mine with a pull-down resistor in the DTX line as suggested by Gabor in his comment. However, I saw — or, to be exact, I heard — no obvious difference. I also ran a few more tests by setting the 3210 to several loopback modes. I tried both ALM1 and ALM2, and was happy to find that there were no audio quality issues at all — at least no issues that my ear could perceive. Similarly, I ran a DLM test by generating a 1-kHz tone, sending that to the board and collecting it back from loopback. Apart from lost packets (on VMWare), this did not show any issues either. However, I will run these tests again, carefully comparing a patched versus an unpatched version of the board, to make sure that the pull-down resistor patch is not really needed. This is very important because it proves that the firmware (including the receipt and transmission of data via the digital DTX/DRX loop) is working perfectly OK, and any potential audio issues can be isolated in the USB layer and up (including the USB communication between the firmware and the host, the  driver, and Asterisk). Hey, this is a test I should have run months ago… Anyway… Finally, here is another great piece of news: I am about to place an order for materials for a bunch of 50 prototypes! Without making any promises yet, prototypes should now be available soon!

Update, May 23: It may seem that I am inactive these days, since I am not posting much. This is not at all the case. The reason I am not posting is that I am doing is a very, very mundane and dull job: I am trying to optimize the cost of my BOM, by looking around at several online electronics shops and retailers. To give an example, a quantity of 50 USB Type A solder-on-board plugs costs about 75 EUR on Digi-Key (Molex), about 100 EUR on mouser (Molex) and about 25 EUR from Farnell. Moreover, in my location, I have to calculate import taxes for merchandise bought from Digi-key. Anyway, in this case Farnell is the clear winner, however this is a very lengthy (and tiresome) work. So far, my per-board BOM cost for 50 prototypes is somewhere between 20 and 30 EUR (this cost does not include assembly and P&P). A friend at a retail shop is currently helping me out to reduce this even further, but I feel the BOM cost won’t drop below 20 EUR/brd. That’s not awfully bad for such a small quantity, and it can drop dramatically at higher quantities. Moreover, there are other optimizations one can make. To give an example, the 0.22uF/100V caps at the output stage cost more than twice as much in a 1812 package than in a 1206 one. I don’t really know why Silabs suggest a 1812 package (I should ask them BTW, or if any reader has gone through this already, a word of advice would be greatly appreciated).  As of now, I have decided not to touch the PCB design and to just try finding the best price for the existing list of materials. Anyway, I am to finalize and place my orders the forthcoming week; until then, probably there are not many news to expect.

Besides hardware, I am also contemplating an optimization for the driver. Here are some details: the driver tends to spend long periods of time in a non-interruptible (IRQ?-)context inside the USB completion callback routines, packetizing and de-packetizing audio data. With the current defaults, this occurs once every four milliseconds for OUT packets plus at the same frequency for IN packets. Ideally, this would occur once every two milliseconds, alternating between OUT and IN callbacks. However, nothing really guarantees this optimal alternating scheduling: the initial submissions that trigger callbacks are executed asynchronously, and it might just as well occur that both IN and OUT callbacks get ready to execute at the exact same millisecond. If more than one boards are plugged, four (or more) callbacks might occur at the same millisecond, and each of those must consume a non-negligible amount of time to packetize/de-packetize data. This might starve the (userspace) Asterisk process from real-time CPU resources and result in “lost” samples (samples that arrive too late) or to bad sound quality. Optimizations include the following: (a) try a preemptive kernel (I am not expecting much there, besides numerous bugs that might show up, both in my and in other people’s code); (b) invoke schedule() at times while doing (de-)packetization and (c) decouple (de-)packetization from the completion callbacks, by adding tasklets to perform the lengthier jobs. I trust that some of the options (a) to (c) above (or a combination thereof) will peform much better than the current driver. Notice that my VMWare testing environment is very demanding anyway in that performance issues show up that would never exist in a native Linux environment. Once I am clear with hardware orders, I might try these optimizations and see what happens. Again, readers who might advise me on the subject are more than welcome!