As I wrote in the blog last week, I was upgrading the code on a fairly large number of switches. And during the upgrade, I did run into a couple of issues and lost a 3500 along with a few 5406′s. I was able to fix and recover all the switches that went down in that change window.
In this blog, I’ll go into details on how I addressed the bad switches and recover them from Monitor ROM and put them back to use.
Since I don’t have a bad switch at the moment I’ll simulate this by booting into Monitor ROM by choosing option 0 when the switch first boots up. And it looks something like this:
Build directory: /sw/rom/build/bmrom(t2g)
Build date: Apr 24 2009
Build time: 23:33:11
Build version: K.12.20
Build number: 24648
Boot Profiles:
0. Monitor ROM Console
1. Primary Software Image
2. Secondary Software Image
Select profile (primary): 0
and here’s what the output looks like when the switch – in this case a 3500yl with 48 copper ports:
ROM Build Directory: /sw/rom/build/bmrom(t2g)
ROM Version: K.12.20
ROM Build Date: 23:33:11 Apr 24 2009
ROM Build Number: 24648
SSC Version: SSC 34 190120060501
MSI Version: 3
CSI Version: 7
Copyright (c) 1995-2005 Hewlett-Packard Company. All rights reserved.
RESTRICTED RIGHTS LEGEND
Use, duplication, or disclosure by the Government is subject to restrictions as set forth in subdivision (b) (3) (ii) of the Rights in Technical Data and Computer Software clause at 52.227-7013.
Hewlett-Packard Company, 3000 Hanover Street, Palo Alto, CA 94303
Enter h or ? for help.
=>
For those of you new to Procurve, the Monitor ROM prompt looks like this =>
When you are in this prompt, you can type h or ? for help:
LAN Monitor Commands
do(wnload) – Download via Xmodem
sp(eed) – Set a new baud rate
h(elp) – Display help screen
? – Display help screen
id(entify) – Print out identification string
jp(jump) – Jump to product code, optional 1-primary, 2-secondary
q(uit) – Exit the monitor
boot – Reboot the system
reset – Reset the system
v(ersion) – Display version information
lsdev – Display I/O device table
fsck – File system check
ls [path] – Terse directory listing
ll [path] – Detailed directory listing
format – Format storage device
blkdump – raw I/O
pwd – Print current working directory
cd – Change directory
mkdir – Make directory
MORE? (Ctl-C to abort)
rmdir – Remove directory
rm – Remove file
cp – Copy file
mv – Move file
cat – Dump file contents
attrib +- – Change the attributes on a file
(a)rchive, (h)idden, (s)ystem, (r)ead-only
=>
Note that the output might vary depending on switch model and code version.
If you press V it will give you version information which you can give you some clue as to what next compatible image is.
Directory: /sw/rom/build/bmrom(t2g)
Build Date: 23:33:11 Apr 24 2009
Version: K.12.20
Build #: 24648
=>
Typically, a switch will boot into Monitor ROM when it has a bad or corrupted flash or when the image is not compatible with the Boot ROM. The way to fix this is to try and put a different image on the switch; one that is compatible with the Boot ROM in the switch. By compatible, I meant the image has to have the same Boot ROM version or one revision up from the current Boot ROM in the switch.
With the software version that I have at the time, the only option I can use to restore the code is through x-modem done over the serial console which can be painfully slow since the a typical serial console connection runs at 9600 baud. To speed things along, I change the baud rate to 115200 by using command speed 115200. When you issue the aforementioned command, you’ll probably see some garbage on the screen since the switch and the computer are now running at different speed. Just change the baud rate on your terminal client to 115200 and hit enter then you’ll get the readable screen back.
To initiate the x-modem transfer, type do for download:
You have invoked the console download utility.
Do you wish to continue? (Y/N)>
The switch will prompt for confirmation. Hit Y then start the transfer. At this point, in your terminal client, start x-modem transfer and pick the firmware you want to send to the switch. You’ll end up with something like this:
1% 59 KB 6 KB/s 00:24:23 ETA 0 Errors
It’ll take a while to send the file. Just remember that you do not have to change the baud rate. 9600 will work just fine but it will take forever to finish. Once the file has been transferred, you can issue the command boot to reboot the switch.
Now that’s how you recover from corrupted flash/wrong code version on Procurve 3500 series switches. On the 5400 series, you can also do the same thing. However, since the 5400 series have removable management module, it can be a little bit easier – you just need some spare modules or some memory card reader.
On the 5400, as I understand it, the boot rom is flashed to NVRAM and the code and configuration files are stored in flash. So what you need to do is making sure NVRAM and flash card have the same or compatible boot rom then the so-called “corrupted flash” will work again. You can do this in two ways. One is to put the correct code on the flash card and two is to flash the correct boot-rom on the management module.
So you can use x-modem transfer to copy code as outline above for the 3500 or find a spare management module and flash compatible boot rom to it, swap out the flash card then put it back to the switch. Since the configuration file is stored on the flash card, you won’t have to reconfigure the switch.
Now here’s something that I’ve been thinking about but haven’t tried yet since I don’t have a card reader handy. The code is stored on a flash card, you can probably use a generic card reader and update it unless Procurve is using some sort of proprietary file system.
I hope with the information above, it can help you restore corrupted flash on Procurve switches. If I have left something out or if you have any tips to share, feel free to leave a comment.
About Brian Dao
Normally I do not learn article on blogs, but I
would like to say that this write-up very pressured me to check out and do it!
Your writing taste has been surprised me. Thank you, quite great post.