Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
Hardware

Hardware Horrors that Firmware Upgrades Would've Fixed? 38

Posted by Cliff from the nobody-is-perfect dept.
Anonymous Coward writes "I just started working for a startup that is developing a new product, which is going to have software bundled with hardware. Our company outsourced the hardware and firmware development. I reviewed the hardware product requirements and I noticed that the hardware will not support firmware upgrades from the PC. I am concerned that once we ship the product, bugs or interoperability issues will appear in the field and we won't have anyway to fix the problem short of a product recall. I have some of the management team convinced we need to change this requirement but not the person who has the authority to make the change. I'm looking for examples of past companies that got bit by a similar mistake and any other items that will help me convince the decision maker." Nobody is perfect, so why do we assume that we can design hardware that is? If it's one thing that our current experiences with software have shown it's that sometimes, an applications may take more than one version before it is perfect. Before, our ability to change hardware coding made getting perfect products out the door important, because recalls were expensive. Today, we have smarter hardware, which can be relatively simple to update. The cost of recalls, however, have not changed. So for what reason would a hardware company balk at making the need for a recall a thing of the past?
This discussion has been archived. No new comments can be posted.

Hardware Horrors that Firmware Upgrades Would've Fixed? More

Hardware Horrors that Firmware Upgrades Would've Fixed?

Comments Filter:

  • iPod (Score:3, Informative)

    by CatatonicBoy ( 1596 ) on Saturday February 02, 2002 @07:50PM (#2943883)
    First thing that comes to mind is the Apple iPod Sleep Issue [apple.com].

  • Direct TV (Score:2, Informative)

    by moheeb ( 228831 )
    Direct TV had this type of problem when their DSS "F" and "H" cards were hacked. They are now on their 3rd generation of smartcards.

    • Re:Direct TV (Score:2, Informative)

      by Jack_of_Hearts ( 135367 )
      No, this has nothing to do with firmware updates. All of DirecTV's updates were made through the satellite, not the smartcards. The smartcard problem involved service theft.
      • the smartcards themselves have microcode that they execute on them. directv ("dave" to dss hackers) periodically sends updates to the cards in order to stomp out service theft - patches are usually to be found within 12 hours though :)
      • This has everything to do with firmware updates.

        The firmware on the smartcard was hacked and was no longer keeping people from getting satellite service for free. DirecTV was forced into recalling the old cards and replacing them with new cards.

  • BIOS .. Palm.. (Score:5, Interesting)

    by josepha48 ( 13953 ) on Saturday February 02, 2002 @08:20PM (#2943998) Journal
    BIOS used to not be all upgradable. It used to be that you would have to replace the chip. Then it dawned on someone to make them software upgradable for bugs and features fixes. Like LBA enhancements.

    Palm pilots (maybe not all, but many and mine) are upgradable. They do this cause they know that software needs to be upgradable. Just about ALL software has bugs.

    If your product goes out and has bugs in it and it causes people loss of data, or worse, you will build yourself a reputation. Sort of like the release of Windows 95 did for Microsoft. No matter how they try they now have a reputation for buggy crashing software. Even if your product is the best on the market if it gets a reputation of bugginess, it will be harder to over come if people have to BUY an upgrade to fix it or BUY a whole new device instead of download bug fixes that makes it worse. While many people will do it they do it till something better comes a long.

    Features are nice, but FIXES are essential to people staying with a product. I stopped useing Microsoft products whenever possible cause I'd rather use a *nix flavor that is less likely to crash on me while typing. This was after my experiences with Win 3.1/95/98 and NT 4.0. I am not impressed enought and do not trust Win2k, Me, or XP. They just don't have the reputation that Sun, BSD, UNIX and Linux have built. I never used a Windows BOX that could stay up for 275 days, but I have seen and used many Sun, BSD, and Linux boxes that were. In fact many of the IT staff people that I have worked with would not support a windows box if you did not reboot it atleast once a day. Also most people I knew or know who do not reboot about once a day end up rebooting when the system crashes.

    So ask your boss, or the person who makes that decision, "Do you want to be a company that works with the consumer to fix the problem and help make their experience with the product better, or do you want to be one that gets a reputation for bad buggy software?"

    • Handsprings don't have Flash ROM, which is why they were initially cheaper. However, you could patch the existing OS with updates, and some handspring modules that required extra features would build them in.

      This caused some problems when the VisorPhone(? maybe it was OmniSky) came out, as it required an OS that was in newer Handsprings, but not older ones. The solution was to give rebates to buy a new handspring.
    • I've got one of the original Palm's... the Pilot 5000 with the calculator bug [aol.com] (1.1-0.1=0.1) in it. 3Com had two possible fixes... one was a software download, which worked great for me. (I'm not sure if it replaced the faulty application, or just a subset of the PalmOS code). The other solution was to replace the mask ROM, which was thoughtfully placed on a removeable card along with all the RAM [twinbrothers.com]. So, if you wanted to upgrade the memory, you'd get the updated ROM, too. (Ah, those were the days... the model 1000 had 128KB and the 5000 had 512KB)

  • My Pentium-133 (Score:3, Interesting)

    by PD ( 9577 ) <slashdotlinux@pdrap.org> on Saturday February 02, 2002 @08:48PM (#2944123) Homepage Journal
    I once had a Pentium 133 that worked properly in all ways except one: When I installed an IDE CD-ROM, the machine could not see the drive. After much digging, I found that the problem was in the AMI BIOS on the motherboard. I bought a new BIOS from Mr. BIOS, and that fixed the problem completely. If that board had a flashable BIOS, the problem could have been solved without changing an IC.

  • Not a problem, but..... (Score:3, Insightful)

    by Patman ( 32745 ) <pmgeahan-slashdot AT thepatcave DOT org> on Saturday February 02, 2002 @09:08PM (#2944196) Homepage
    Voyetra/Turtle Beach's Audiotron home MP3 player has an easily flashable firmware. The system ships as just a player, but the firmware adds functionality such as advanced management and web-based control. Point out to your manager that it's not just for problems - it can be used for features as well!

  • CD-ROM drives (Score:3, Informative)

    by Snowfox ( 34467 ) <snowfox@[ ]wfox.net ['sno' in gap]> on Saturday February 02, 2002 @10:13PM (#2944412) Homepage
    You would think those $40 CD-ROM drives would be the last place you would see upgrade-capable firmware.

    Not so.

    Many of the 12x and 16x units wouldn't read CDRs, which would have made them near worthless in today's world. For most of them, the fix was as simple as slowing the speed and trying another pass at reading before giving up.

    For many brands, a flash upgrade was all it took to fix these and give them value again. The upgrade was made available to consumers who suddenly had brand loyalty for what's normally a pretty ambigously branded piece of hardware.

    For many other brands, the units became bargain bin fodder and left a lot of consumers pissed off at what they thought was broken hardware.

  • I've worked on a number of Netopia [netopia.com] routers that have required firmware upgrades. We'll be installing them for clients who have strange setups that aren't common for most people, and on occasion the problems we run into are solved by a firmware upgrade. Netopia's good at posting upgrades to their FTP server, even with products that have been out for a while.


    Word of caution though, and I'm sure you've heard it before. Don't upgrade the firmware unless you are having problems. One time I updated the firmware on one of these routers, and ended up breaking more things than I was trying to fix! I called up Netopia's support line, and apparently this upgrade was a buggy one, it had only been posted on the FTP site for a few hours when they found the bugs. Bad luck for me trying to update during that time though. But hey, those are the breaks.

  • This was a combination of user error and firmware issues, but we had a disk array lose its mind (and 30 GB of RAIDed data) once. While the tech shouldn't have turned the array off, if there weren't a firware bug, it would have flushed the cache and we wouldn't have lost the data. I would up spending a weekend flipping bits to get back half the data. Not Happy.
    • I saw the same thing happen to a Hitachi storage unit. From what I can recall, a lot of the HP storage is relabeled Hitchi with slightly different firmware.
  • that you can update the firmware.. toshiba, sony, ect.. their DVD players all accept firmware updates reason being is that newer movies may or may not play properly as the software gets newer. problems range from lockups to pixelation, garbage on the screen. im an electronics tech, and I see this daily. so, firmware updates are VERY important. on older units, we have to loterally remove the EEPROM and replace it. new ones, an interface and a serial cable is all thats needed. of course, only authorized service centers have this hardware/software. but, your correct... the ability to update the firmware will be VERY helpful. and it makes the life of the tech easier :) hope this helps you.

  • Every ISP using Cisco CPE's (Score:3, Informative)

    by cybersquid ( 24605 ) on Sunday February 03, 2002 @12:53AM (#2944917) Homepage
    Well, here is an example of being saved by a firmware update.

    Back around August 2001, that famous MSTD [everything2.com], the CodeRed worm was swarming across the Internet. One side effect of it's probing behavior was to trigger a bug in certain models of Cisco DSL modems. The result was a crashed modem.

    The user could power cycle the modem, but it would die again shortly when their neighbor's infected system probed them. This was a catastrophe for the ISP's involved.

    This effected many people, more than a million I believe.

    Cisco put out a corrective CD-ROM that reflashed the CPE with fixed firmware. If this had not been possible, Cisco would probably have ended up paying to replace all those modems. Running off some CD-ROMs was a lot cheaper.

  • The firmware upgrades are what saved this poduct [linksys.com]/

    Read the readme.txt for a long history of bugs that have been fixed through firmware upgrades - originally I had problems with it, and it was a firmware upgrade that fixed it - the saving grace for this product.

    Having firmware upgrades for a product is a very prudent thing to do. Anybody that doesn't think so is arrogant.

    • And in the startup configuration, you can "upgrade" the firmware without a password from the WAN side, using TFTP.

      Alterable firmware can create security problems. So far, Microsoft products have presented such a vulnerable monoculture that few people have bothered to write attacks on firmware. That may change once Microsoft tightens up their mess.

  • Upgradeable firmware extended Viking life (Score:4, Informative)

    by clem.dickey ( 102292 ) on Sunday February 03, 2002 @03:00AM (#2945173)
    (Saga which I read in Science magazine many years ago.)

    The 1975 Viking Mars lander was expected to last only a few months on the Mars surface; battery life was the limiting factor. The battery lasted longer than expected, but eventually the Sun would come between Earth and Mars. With the lander fully powered the battery would be dead by the time Earth came back into view.

    NASA (or maybe it was JPL) thought of reprogramming the Viking controller to power down, wait a few months, then power back up. (The power-up had to be automatic; in power-down mode there was no communication with Earth.)

    Viking had reprogrammable firmware, but only for pre-flight programming. Reprogramming during the mission hadn't been anticipated, so the diagnostic bus through which the ROM was reprogrammed was removable. There was no record of whether the Viking which NASA had sent to Mars had that bus or not! Nor was there a way to detect bus presence.

    On the chance that the bus was installed, new code was tested on an Earth-bound copy of the Viking which had the bus, then uploaded. The Mars lander did have the bus, the code worked, and NASA got several additional months of operation from Viking.
  • satellites are kindof an extreme case of hard-to-service-in-the field. Especially when the one-way shipping cost = 10x the cost of the satellite (we did small sats). So, getting uploadable code right was a top priority. Radiation could damage most of our memory and upset (reset) our processor. The EPROM held multiple versions of our main software. A small amount of expensive rad-hard PROM code performed checksums on the EPROM and executed an uncorrupted copy. We didn't have FLASH memory, so any new code was stored in RAM, which was even more susceptable to bit hits than the EPROM. Getting the code up to the satellite was something we had to plan well... we had a 9600 baud link that we could use only ~7 minutes out of each 90 minute orbit. At this rate, uploading a new version of code would take quite a while and would have to be performed every time a radiation-induced reset occured. It was going to be a maintenance/reliablity nightmare!! Unfortuantly, our rocket blew up and we never got to try out the satellite.

  • I have a Garmin eMap [garmin.com]. The flash upgrade feature is great, I probably would've had to buy a whole new unit to get the kind of features they've packed into the latest firmware update.

    The list of corrections and features they've added to this thing is amazing [garmin.com]. The eMap was useful before, now it's downright a necessity when I'm traveling.

  • How much did it cost Intel to replace all those P5-60s and -90s back in 1994/95?

    Perhaps that's why they've introduced microcode update functionality (typically done by your BIOS, but there's stuff in the Linux kernel to allow you to do it from the OS).

    --

    • Actually, IBM was doing this a long time ago. Owing to the cost of their hardware, companies started producing hardware that did the same job as IBM kit, but without the ability to upgrade the microcode. They ran faster (I believe), and were much cheaper. However, IBM put a stop to it, I think by changing the microcode and the OS binaries. Then the clone kit wouldn't run the new software, and that was that.
      • Yup, I seem to recall that Amdahl were bitten by this when 360 (?) mainframes were all the rage. Before my time (at least professionally ;-) so the memory is a little hazy...

        --
    • About $500 million is the official figure.
  • The Rio 500 firmware started out at 1.0. The last released firmware was 2.15, and there were plenty of releases inbetween - go figure. Apart from fixing a few battery bugs, they also added support for 64Mb Smart Media cards and ID3 tags, both extremely welcome.

    On the other hand, the BIOS for my motherboard requires a chip swap to upgrade. Fortunately, I haven't discovered a serious reason to do this yet, although there are some annoying DMA issues I'd rather fix. But am I gonna mess around ordering and fitting a new chip or will I simply upgrade the board (to another brand)? You got it...

    Ade_
    /
  • Ok, anon coward. Ask me.
    I'm not going to name names now, but I used
    to work for a company that developed embedded
    hardware devices. At least they had a method for
    upgrading these devices from the host PC, but
    some versions were old or borked because they
    were development versions. These units were
    essentially not field-upgradeable. a complete
    "pita" and PR disaster when these things needed to be fixed. feel free to email for specifics.
    At this point, I would not design any embedded
    system without a safe way to do field upgrades.
    (What the hell is wrong with the lameness filter?)

    Greg

    galaiama@saunasamaaoakae.org

  • On the down side... You can kill the product. (Score:3, Insightful)

    by grnbrg ( 140964 ) <<slashdot> <at> <grnbrg.org>> on Sunday February 03, 2002 @02:14PM (#2946829)
    Flash is great, but make sure that it is failsafe.

    Example -- I had 2 USR Courier modems in the mid-90s that were 'flash upgradeable'... Once the V90 standard was stable, I flashed one of them.

    And killed it.

    The modem was supposed to be flashable, and I did everything right, but USR had got the hardware wrong. They replaced both modems at their cost, and both the new modems flashed correctly.

    There are also frequently warnings on motherboard flash programs and Palm flash programs to this effect -- if you screw up the flash, you will have a product that you *cannot* use, and must send in/replace to get functioning again.

    With that in mind, make sure your product either has a flash loader, or default software image in ROM that can be accessed if the flash image is corrupt, either automatically, or by a jumper. Otherwise you may end up with angry customers who have upgraded paperweights.

    Brian.
  • Dell's recent laptops/desktops have flash bios updates that come out every few weeks. They fix bugs, but also introduce new ones. Be careful with this type of thing! Just because it's easy to make an update doesn't mean that it's okay to skimp on testing because a fix is easy to shove to the customer.

    It's fairly obvious that they failed to test the Inspiron 8100 A08 Bios in a system with no floppy drive (e.g. two battery configuration). Once the machine tries to access the floppy drive, the hard drive activity light (shared with the floppy, actually) will not turn off until you suspend or power down. Of course, an antivirus program initializing when you start Windows will access the floppy (that isn't plugged in) and cause the light to turn on forever. While this isn't an issue that causes BSOD (this update actually fixed a few of those bugs), it's definitely an annoyance.
  • I'd start putting my resume out immediately. To do otherwise is a bet that everyone involved will do everything right the first time.

    If you have bosses stupid enough to bet the company on this, their business and that of the unfortunates who invested, but you have no reason to bet your career with them.

    Unless you've got a shitload of cash and the company's got really cool technology, in which case your best move is to wait. . . and buy it at fire sale prices, rebrand the product, and build it right this time.

  • Some of the first PowerPC Performas and Powermacs (circa 95) had issues with the firmware and ROMs forcing apple to replace boards in *alot* of machines. It was no doubt a tremendous pain in the rear, and proably cost them alot of cash and PR standing.
  • I don't know much about the details of any of these (other than how easy it was to kill a Pentium with the f00f bug), but when I do cat /proc/cpuinfo, it mentions all of them. Presumably the Linux kernel has workarounds for them all.
  • It's really pretty tricky to design an embedded system so that re-programming the Flash on board is easy enough for lusers to do, and yet it will never, ever get changed by accident. PC's are probably the easiest cast, since they come with built-in serial comm, a full-featured OS, and user interface through a full-sized display, keyboard, and mouse -- yet I hear of accidents happening in re-flashing and leaving the MB dead until the chip is physically replaced.

    Now think of doing this with a CD drive, which has no display other than a few LED's, no keyboard other than a few function keys, and no communications capability...
  • If you have any question about whether or not in-circuit reflashability is a good idea, look at the auto industry. Very soon, every part in the car that has a connection to the vehicle bus will be specced to receive software upgrades from the bus. And why not? Have you considered how expensive it is to pull a million modules from inside the dashes, beneath the seats or even from under two other things inside the engine compartment, re-flash them, and put them back?

    The economics of your company are probably quite a bit different from the auto industry; your volume is probably several orders of magnitude less, to name one thing. But you have to consider the loss of goodwill if customers have to pull hardware and ship it to you for firmware fixes, instead of taking 20 minutes to download and install a new patch. You might also consider the benefits of being able to sell firmware functionality upgrades for units already installed; the customers will love you for teaching their old dog a new bunch of tricks.

Slashdot Top Deals

The Tao is like a glob pattern: used but never used up. It is like the extern void: filled with infinite possibilities.

Close