Monday, March 3, 2008

Manually upgrading PC-BSD v1.4 to using FreeBSD 7 underneath

It can be done although I don't actually recommend it lol, I did this solely for fun. Here is what I recored in the ~/fun-log file on my Samba server. A copy of my kernel configuration and a screen shot of the first GUI enabled boot is at the end of the post:


------------------------------------------------------------------
Enabled SSH login to my test machine, since I don't use the default port 22 for SSH I had to open ssh_config and sshd_config in /etc/ssh/ and uncomment the 'Port' line and change the port number.

A quick restart of the SSH Daemon from a root shell:

/etc/rc.d/sshd restart

I also disabled the firewall (pf) since I was in no mood to sort it out or port my laptops pf.config over.

and from my own shell

ssh Terry@xxx.xxx.xxx.xxx # s/xxx\.xxx\.xxx\.xxx/server ip/g

I accepted the finger print and logged in, then ssh'd to this machine from that ssh connection doing like wise.

I'd recommend taking back ups of the system before proceeding, since this is a test machine I can skip it. I would suggest using either tar or dump in conjunction with a mounted recovery partition (or other storage device) or SSH if you have another system to use for storing the backups. Some people prefer Optical disks (cd/dvd) but I only use these for perm. backups.


rebooted with the FreeBSD 7.0-RELEASE install cd in the drive and chose the upgrade option from sysinstall. I chose to upgrade 'custom' and selected all sets except the profiled libraries and source code: every thing worked perfect except that X.Org failed to upgrade due to already installed packages, unexpected yes but no problem as I was not planning on using X until after KDE was upgraded.

Completed the upgrade and chose the 'Fixit' option in the main menu dropping to a shell on ttyv4 which I used to edit /etc/ttys and disable the start of X.Org on boot up.

Located the ttyv4 line that starts /PCBSD/bin/pdm on an xterm and change the 'on' to an 'off'. Saved the file and gave a 'reboot'.


I forgot that FreeBSD's GENERIC kernels detect my hard drives wrong because the kernel uses the ATA_STATIC_ID option. I have a SATA drive for ad0 and with ATA_STATIC_ID in the kenrel config it detects ad0 as ad4, PC-BSD's kernel builds have the 'right' option commented out to fix that so I had to manually select a root disk at boot:

ufs:ad4s3

While I could easily fix this with changing the lines in /etc/fstab that would defeat the point of merging PC-BSD's SMP Kernel configuration with FreeBSD's GENERIC kernel. Which I could have done before rebooting any way.


I Inserted my install disk and ran /usr/sbin/sysinstall, using the 'Configure' option I chose to install the source code for every thing through the distribution sets. As far as I know you need the full system source to build a kernel.


I set PACKAGEROOT and did a pkg_add of lynx so I could view the installed version of the FreeBSD handbook without reading HTML source code. I like to have access to the handbook when building a kernel so that I can check if any new notes have been made in the section on the kernel configuration file. I also did a pkg_add of the mg editor, much smaller then vim but easier for me to work with then FreeBSDs /usr/bin/vi when I wish to view two files at once.

mkdir ~/kernel-config; cd ~/kernel-config
cp /PCBSD/conf/PCBSD-SMP.i386 ./
cp /usr/src/sys/i386/conf/GENERIC ./
pkg_add -r mg
 ...
vi ~/.mg # set a few options I like for emacs

Then I used MicroGNUEmacs (mg) to merge PCBSD-SMP.i386 into GENERIC saving it as mykernel with the ^x^w command, the basics of using mgh you can find in the man page; if you choose to use mg but don't know emacs. In point of fact, I have not used emacs regularly to edit files in over 2 years!

touch ./mykernel; ln -s /root/kerne-config/mykernel /usr/src/sys/i386/conf/mykernel; mg ./mykernel
cd /usr/src
make buildkernel KERNCONF=mykernel

You know, I wish I remembered to use the -j 8 option to make... Because last time I did that on my Pentium D it only took ~45 minutes to compile a kernel on 6.1-RELEASE or so.

Build started at: Sun Mar 2 21:19:14 UTC 2008
Build finished at: Sun Mar 2 21:40:51 UTC 2008

And to top it off, when I got up to do a few chores when I got back the console 'screen saver' kicked in. Which just happened to be the PC-BSD splash screen that comes up when ever the system boots kicked in during my kernel compile. This appears to have been some thing I broke with the upgrade, unless there is a difference because my laptop uses a Beastie image for the console screen saver.


I'm standing here like WTF? Did I crash or are they joking. Sure enough the system was playing a mean joke on me haha. And I know the system shouldn't go down during a buildkernel even with what I've been 'doing' to the system without some serious problems. Right now it looks like the NIC drivers are compiling so it shouldn't take to much longer.

While I wait, I'll start merging the configuration files in /etc with those in /etc/upgrade.

A quick look to see if there are any config files from PC-BSD with pcbsd mentioned in them:

find /etc/ -type f -exec grep -ni 'pcbsd|pc-bsd' '{}' \;

Of course they are *never* so nice as to denote what files are explicitly changed from the base FreeBSD installs 8=). Used lynx to browse PC-BSD's SVN Repository online, looking at the system overlay in the 1.4 branch. I don't see any any thing here that should effect me seriously -- if any one tries this after me, check PC-BSDs SVN -> pcbsd/branches/1.4/system-overlay/etc (some thing like that) and look at the commit messages, if you see some thing that might effect you. Carefully compare the relevant files in /etc/ and /etc/upgrade unless like me, there is nothing you can seriously break that you will ever want to fix later.

There are two ways of doing this part, manual and mergemaster. If you have never done a buildworld/installworld before read the manual page for mergemaster and run it in a more user friendly mode. I usually do this any way because I use mergemaster only a few times a year at the most.

Some switches to mergemaster you might want to look up are: -a, -i, -v, -U

If you are going manually, I suggest you move the files from /etc/upgrade to /etc/ but pay special attention to files such as group and rc.local -> You don't want to be unable to login as any thing but root on your reboot now do you? Hehe.

mv /etc/upgrade /root/etc-upgrade
mergemaster -viU
 ...

I installed most files but merged several others, among them:

ssh_conf and sshd_conf I merged to keep my port settings

group I merged to keep the various groups PC-BSD has setup for HAL'd and friends as well as having my user in wheel, operator, and its own group ;-)

login.conf because I use 'blf' for the pass word format rather then the default md5. At the end of my mergemaster'ing I was happy to see that mergemaster was already a step ahead of me and promped for the login.conf database to be rebuilt for fool proof safeties sake:

*** You installed a login.conf file, so make sure that you run
    '/usr/bin/cap_mkdb /etc/login.conf'
     to rebuild your login.conf database

    Would you like to run it now? y or n [n] y
    Running /usr/bin/cap_mkdb /etc/login.conf

Some files that I chose to delete, the extra rc scripts which may have broken one of the Intel Wifi drivers. Also I decided not to install the *new* printcap file -> which if I had installed would have overwritten any printer setup I have done on my test machine.

One nice advantage is I have PC-BSD's kernel stored in /boot/kernel.prev from the FreeBSD upgrade proceedure and the GENERIC FreeBSD 7.0-RELEASE kernel in /boot/kernel.old after installing my newly compiled custom kernel.

For extra safety, I'll archive these in roots home directory with more normal names in case I need them later:

tar -cf - /boot/kernel.old | bzip2 -9 > /root/kernel-FreeBSD7.0-RELEASE.g
eneric.tar.bz2
tar -cf - /boot/kernel.prev | bzip2 -9 > /root/kernel-PCBSD1.4.2.smp.tar.
bz

I don't know how to get BSD tar to adjust the compression level for gzip/bzip2 (-z,-j), assuming there is a switch for it. So I redirected tar's output to bzip2 directly to enable maximum compression.


Sadly, on reboot my system would lock up when ever the kernel tried to probe /dev/da3. No matter how many times I try to boot it, it locks up on this. I booted the Windows XP installation on the machine, rebooted into PC-BSD again and it boots perfectly -- This is a *standing* issue with this test machine.


Often FreeBSD will lock up during boot while probing my (internal) USB Card Reader when it gets to the memory stick slot and refuse to boot until I have booted another operating system on the machine. I have had this problem since FreeBSD 6.0-RELEASE, never tried any earlier ones.


Yes I hate this computer.




But otherwise my system however works bloody perfect :-)

pcbsd# uname -ai
FreeBSD pcbsd 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Mar 2 16:27:13 EST 2008
root@pcbsd:/usr/obj/usr/src/sys/mykernel i386 TERRYP


Now to get X.Org and KDE back online, installed the ports tree from the FreeBSD 7.0-RELEASE CD-ROM, new enough for my needs. While that goes, lets back up a few things of interest:

tar -czf /root/old-ports.tgz /usr/local/etc /usr/local/share/{icons,config,config.kcfg,apps,applnk}
tar -cjf /root/linux-fc4.tar.bz2  /compat

These files are essentially our KDE icons and a few directories we might want to keep handy for later. Also I chose to backup the /compat directory that houses the linux files.

Now for the portupgrade business:

After telling the pkgdb what to do with it self, I checked the help and gave this a go after I got tired of the interactive:

pkgdb -fFi

The one thing I hate about dealing with portupgrade and friends on FreeBSD is the packages database -- it's a royal pain in the arse to deal with when you are not in the mood! After sorting out all of the crap that pkgdb had to shout at including the mother ****ing **** load of fonts mentioned in the sale dependencies I know I will __never__ do this again.


In fact, if I ever go back to maintaining a standard FreeBSD system for my workstation I'll do this my way -- use a list of programs needed, and do upgrades with out mucking with pkgdb and it's friends lol -> less trouble I think to do it by hand the way my mind is thinking of it now hehe.


Now to fetch every thing needed before we start and prefer binary packages where possible to save on some compile time. -> Warning!!! Not my reccomendation, done here out of my lack of concern for breaking the test machine.

portupgrade -FDParR

Now to upgrade every thing preferring packages, go to splitsvile and read the log later.

portupgrade --batch -faPrR --results-file /root/portupgrade.log


Whether -a impiles -rR or not I don't know, never really read the source code that much but I'm used to using all three switches at once for this. Portupgrade also crashed about 3 times during the installation, mostly from upgrading ruby and missing stuff in /tmp. It also fragged /var/db/pkg/pkgdb.db once or twice and shouted at /usr/ports/INDEX-7.db's format. So I had to run the command several times taking about a day and a half to complete the entire operation as described here.


Now I remember why I don't use portupgrade or mass software upgrades a lot on my stable machines.

I had to create a new xorg.conf so I did an

X -configure
X -config /root/xorg.conf.new

and it worked, a reboot and I was soon greated by KDE 3.5.8 running on X.Org 7.3 ;-)


I know I broke the PBReg program so I would expect most other custom programs that PC-BSD has needs a recompile from the source code. I also used as many packages as possible during the upgrade procedure to save time, not what I normally do on FreeBSD but it actually worked well enough. Tomorrow after work I'll test out a few of the PC-BSD programs and the Firefox PBI I have installed on the test machine, and installing a PBI to see if any thing survived without a recompile needed. I also want to test the Linux ABI's new abilities and see what happens when installing linux-flock binaries.

------------------------------------------

Screen shot:

Free Image Hosting at www.ImageShack.us


Here is a copy of my kernel configuration:

# My PC-BSD v1.4.2 + FreeBSD 7.0-RELEASE kernel configuration

cpu  I686_CPU
ident  TERRYP

# To statically compile in device wiring instead of /boot/device.hints
#hints  "GENERIC.hints"  # Default places to look for devices.

makeoptions DEBUG=-g  # Build kernel with gdb(1) debug symbols

#options  SCHED_4BSD  # 4BSD scheduler
options  SCHED_ULE  # ULE scheduler instead ;-)

# options from PC-BSD that I'll keep handy
options  LIBICONV
options  LIBMCHAIN
options  CD9660_ICONV
options  MSDOSFS_ICONV
options  NTFS
options  NTFS_ICONV
options  UDF
options  UDF_ICONV
options  GEOM_UZIP
options  DEVICE_POLLING
device  iwi  # intel wireless adapters
device  ipw  # intel wireless adapters

#### I want to use PF.
device  pf
device  pflog
device  pfsync

options  ALTQ
options  ALTQ_CBQ
options  ALTQ_RED
options  ALTQ_RIO
options  ALTQ_HFSC
options  ALTQ_CDNR
options  ALTQ_PRIQ
options  ALTQ_NOPCC

# Memory card drivers I want to test:
device  mmc
device  mmcsd

# options in GENERIC
options  INET   # InterNETworking
options  INET6   # IPv6 communications protocols
options  SCTP   # Stream Control Transmission Protocol
options  FFS   # Berkeley Fast Filesystem
options  SOFTUPDATES  # Enable FFS soft updates support
options  UFS_ACL   # Support for access control lists
options  UFS_DIRHASH  # Improve performance on big directories
options  UFS_GJOURNAL  # Enable gjournal-based UFS journaling
options  MD_ROOT   # MD is a potential root device
options  NFSCLIENT  # Network Filesystem Client
options  NFSSERVER  # Network Filesystem Server
options  NFS_ROOT  # NFS usable as /, requires NFSCLIENT
options  MSDOSFS   # MSDOS Filesystem
options  CD9660   # ISO 9660 Filesystem
options  PROCFS   # Process filesystem (requires PSEUDOFS)
options  PSEUDOFS  # Pseudo-filesystem framework
options  GEOM_PART_GPT  # GUID Partition Tables.
options  GEOM_LABEL  # Provides labelization
options  COMPAT_43TTY  # BSD 4.3 TTY compat [KEEP THIS!]
options  COMPAT_FREEBSD4  # Compatible with FreeBSD4
options  COMPAT_FREEBSD5  # Compatible with FreeBSD5
options  COMPAT_FREEBSD6  # Compatible with FreeBSD6
options  SCSI_DELAY=5000  # Delay (in ms) before probing SCSI
options  KTRACE   # ktrace(1) support
options  SYSVSHM   # SYSV-style shared memory
options  SYSVMSG   # SYSV-style message queues
options  SYSVSEM   # SYSV-style semaphores
options  _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options  KBD_INSTALL_CDEV # install a CDEV entry in /dev
options  ADAPTIVE_GIANT  # Giant mutex is adaptive.
options  STOP_NMI  # Stop CPUS using NMI instead of IPI
options  AUDIT   # Security event auditing

# To make an SMP kernel, the next two lines are needed
options  SMP   # Symmetric MultiProcessor Kernel
device  apic   # I/O APIC

# CPU frequency control
device  cpufreq

# Bus support.
device  eisa
device  pci

# Floppy drives
device  fdc

# ATA and ATAPI devices
device  ata
device  atadisk  # ATA disk drives
device  ataraid  # ATA RAID drives
device  atapicd  # ATAPI CDROM drives
device  atapifd  # ATAPI floppy drives
device  atapist  # ATAPI tape drives
#### XXX:
####       This option must be commented out for SATA drives to be
####       detected properly, e.g. ad0 not ad4 on boot up!
#options  ATA_STATIC_ID # Static device numbering

# SCSI Controllers
device  ahb  # EISA AHA1742 family
device  ahc  # AHA2940 and onboard AIC7xxx devices
options  AHC_REG_PRETTY_PRINT # Print register bitfields in debug
     # output.  Adds ~128k to driver.
device  ahd  # AHA39320/29320 and onboard AIC79xx devices
options  AHD_REG_PRETTY_PRINT # Print register bitfields in debug
     # output.  Adds ~215k to driver.
device  amd  # AMD 53C974 (Tekram DC-390(T))
device  hptiop  # Highpoint RocketRaid 3xxx series
device  isp  # Qlogic family
#device  ispfw  # Firmware for QLogic HBAs- normally a module
device  mpt  # LSI-Logic MPT-Fusion
#device  ncr  # NCR/Symbios Logic
device  sym  # NCR/Symbios Logic (newer chipsets + those of `ncr')
device  trm  # Tekram DC395U/UW/F DC315U adapters

device  adv  # Advansys SCSI adapters
device  adw  # Advansys wide SCSI adapters
device  aha  # Adaptec 154x SCSI adapters
device  aic  # Adaptec 15[012]x SCSI adapters, AIC-6[23]60.
device  bt  # Buslogic/Mylex MultiMaster SCSI adapters

device  ncv  # NCR 53C500
device  nsp  # Workbit Ninja SCSI-3
device  stg  # TMC 18C30/18C50

# SCSI peripherals
device  scbus  # SCSI bus (required for SCSI)
device  ch  # SCSI media changers
device  da  # Direct Access (disks)
device  sa  # Sequential Access (tape etc)
device  cd  # CD
device  pass  # Passthrough device (direct SCSI access)
device  ses  # SCSI Environmental Services (and SAF-TE)

# RAID controllers interfaced to the SCSI subsystem
device  amr  # AMI MegaRAID
device  arcmsr  # Areca SATA II RAID
device  asr  # DPT SmartRAID V, VI and Adaptec SCSI RAID
device  ciss  # Compaq Smart RAID 5*
device  dpt  # DPT Smartcache III, IV - See NOTES for options
device  hptmv  # Highpoint RocketRAID 182x
device  hptrr  # Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx
device  iir  # Intel Integrated RAID
device  ips  # IBM (Adaptec) ServeRAID
device  mly  # Mylex AcceleRAID/eXtremeRAID
device  twa  # 3ware 9000 series PATA/SATA RAID

# RAID controllers
device  aac  # Adaptec FSA RAID
device  aacp  # SCSI passthrough for aac (requires CAM)
device  ida  # Compaq Smart RAID
device  mfi  # LSI MegaRAID SAS
device  mlx  # Mylex DAC960 family
device  pst  # Promise Supertrak SX6000
device  twe  # 3ware ATA RAID

# atkbdc0 controls both the keyboard and the PS/2 mouse
device  atkbdc  # AT keyboard controller
device  atkbd  # AT keyboard
device  psm  # PS/2 mouse

device  kbdmux  # keyboard multiplexer

device  vga  # VGA video card driver

device  splash  # Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device  sc

device  agp  # support several AGP chipsets

# Power management support (see NOTES for more options)
#device  apm
# Add suspend/resume support for the i8254.
device  pmtimer

# PCCARD (PCMCIA) support
# PCMCIA and cardbus bridge support
device  cbb  # cardbus (yenta) bridge
device  pccard  # PC Card (16-bit) bus
device  cardbus  # CardBus (32-bit) bus

# Serial (COM) ports
device  sio  # 8250, 16[45]50 based serial ports
device  uart  # Generic UART driver

# Parallel port
device  ppc
device  ppbus  # Parallel port bus (required)
device  lpt  # Printer
device  plip  # TCP/IP over parallel
device  ppi  # Parallel port interface device
#device  vpo  # Requires scbus and da

# If you've got a "dumb" serial or parallel PCI card that is
# supported by the puc(4) glue driver, uncomment the following
# line to enable it (connects to sio, uart and/or ppc drivers):
#device  puc

# PCI Ethernet NICs.
device  de  # DEC/Intel DC21x4x (``Tulip'')
device  em  # Intel PRO/1000 adapter Gigabit Ethernet Card
device  ixgb  # Intel PRO/10GbE Ethernet Card
device  le  # AMD Am7900 LANCE and Am79C9xx PCnet
device  txp  # 3Com 3cR990 (``Typhoon'')
device  vx  # 3Com 3c590, 3c595 (``Vortex'')

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device  miibus  # MII bus support
device  bce  # Broadcom BCM5706/BCM5708 Gigabit Ethernet
device  bfe  # Broadcom BCM440x 10/100 Ethernet
device  bge  # Broadcom BCM570xx Gigabit Ethernet
device  dc  # DEC/Intel 21143 and various workalikes
device  fxp  # Intel EtherExpress PRO/100B (82557, 82558)
device  lge  # Level 1 LXT1001 gigabit Ethernet
device  msk  # Marvell/SysKonnect Yukon II Gigabit Ethernet
device  nfe  # nVidia nForce MCP on-board Ethernet
device  nge  # NatSemi DP83820 gigabit Ethernet
#device  nve  # nVidia nForce MCP on-board Ethernet Networking
device  pcn  # AMD Am79C97x PCI 10/100 (precedence over 'le')
device  re  # RealTek 8139C+/8169/8169S/8110S
device  rl  # RealTek 8129/8139
device  sf  # Adaptec AIC-6915 (``Starfire'')
device  sis  # Silicon Integrated Systems SiS 900/SiS 7016
device  sk  # SysKonnect SK-984x & SK-982x gigabit Ethernet
device  ste  # Sundance ST201 (D-Link DFE-550TX)
device  stge  # Sundance/Tamarack TC9021 gigabit Ethernet
device  ti  # Alteon Networks Tigon I/II gigabit Ethernet
device  tl  # Texas Instruments ThunderLAN
device  tx  # SMC EtherPower II (83c170 ``EPIC'')
device  vge  # VIA VT612x gigabit Ethernet
device  vr  # VIA Rhine, Rhine II
device  wb  # Winbond W89C840F
device  xl  # 3Com 3c90x (``Boomerang'', ``Cyclone'')

# ISA Ethernet NICs.  pccard NICs included.
device  cs  # Crystal Semiconductor CS89x0 NIC
# 'device ed' requires 'device miibus'
device  ed  # NE[12]000, SMC Ultra, 3c503, DS8390 cards
device  ex  # Intel EtherExpress Pro/10 and Pro/10+
device  ep  # Etherlink III based cards
device  fe  # Fujitsu MB8696x based cards
device  ie  # EtherExpress 8/16, 3C507, StarLAN 10 etc.
device  sn  # SMC's 9000 series of Ethernet chips
device  xe  # Xircom pccard Ethernet

# Wireless NIC cards
device  wlan  # 802.11 support
device  wlan_wep # 802.11 WEP support
device  wlan_ccmp # 802.11 CCMP support
device  wlan_tkip # 802.11 TKIP support
device  wlan_amrr # AMRR transmit rate control algorithm
device  wlan_scan_ap # 802.11 AP mode scanning
device  wlan_scan_sta # 802.11 STA mode scanning
device  an  # Aironet 4500/4800 802.11 wireless NICs.
device  ath  # Atheros pci/cardbus NIC's
device  ath_hal  # Atheros HAL (Hardware Access Layer)
device  ath_rate_sample # SampleRate tx rate control for ath
device  awi  # BayStack 660 and others
device  ral  # Ralink Technology RT2500 wireless NICs.
device  wi  # WaveLAN/Intersil/Symbol 802.11 wireless NICs.
#device  wl  # Older non 802.11 Wavelan wireless NIC.

# Pseudo devices.
device  loop  # Network loopback
device  random  # Entropy device
device  ether  # Ethernet support
device  sl  # Kernel SLIP
device  ppp  # Kernel PPP
device  tun  # Packet tunnel.
device  pty  # Pseudo-ttys (telnet etc)
device  md  # Memory "disks"
device  gif  # IPv6 and IPv4 tunneling
device  faith  # IPv6-to-IPv4 relaying (translation)
device  firmware # firmware assist module

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device  bpf  # Berkeley packet filter

# USB support
device  uhci  # UHCI PCI->USB interface
device  ohci  # OHCI PCI->USB interface
device  ehci  # EHCI PCI->USB interface (USB 2.0)
device  usb  # USB Bus (required)
#device  udbp  # USB Double Bulk Pipe devices
device  ugen  # Generic
device  uhid  # "Human Interface Devices"
device  ukbd  # Keyboard
device  ulpt  # Printer
device  umass  # Disks/Mass storage - Requires scbus and da
device  ums  # Mouse
device  ural  # Ralink Technology RT2500USB wireless NICs
device  rum  # Ralink Technology RT2501USB wireless NICs
device  urio  # Diamond Rio 500 MP3 player
device  uscanner # Scanners

# USB Ethernet, requires miibus
device  aue  # ADMtek USB Ethernet
device  axe  # ASIX Electronics USB Ethernet
device  cdce  # Generic USB over Ethernet
device  cue  # CATC USB Ethernet
device  kue  # Kawasaki LSI USB Ethernet
device  rue  # RealTek RTL8150 USB Ethernet

# FireWire support
device  firewire # FireWire bus code
device  sbp  # SCSI over FireWire (Requires scbus and da)
device  fwe  # Ethernet over FireWire (non-standard!)
device  fwip  # IP over FireWire (RFC 2734,3146)
device  dcons  # Dumb console driver
device  dcons_crom # Configuration ROM for dcons

1 comment:

  1. Originally posted on my Live Journal:

    subject: good work
    by: (Anonymous) at 2008-03-04 04:15 pm (UTC)
    comment: Nicely documented, especially the snags.
    Way to go. :)

    ReplyDelete