Fibre Channel storage with HBA target mode

General considerations

FC targets usually come in form of (mostly) HDD's with a Fibre Channel interface which are used in suitable enclosures. Also tape drives with FC interface can be found, but these are rare compared to the ones with SCSI interface. In most cases they are used in huge tape librarys. While the FC HDD enclosures are nice for some applications (especially where RAID or JBOD functions are necessary or you want to achieve high speeds) they are also loud, big and power-hungry. Not what everybody wants in his room and running 24/7...

But there are other ways to attach your devices to FC:

• SCSI to FC bridges/routers
• FC HBA's with target mode drivers.

The latter option is a very interesting one because it lets you create an inexpensive, silent FC storage solution with relative good performance. It also includes the functionality of the aforementioned SCSI-FC bridge (depending of the software you will use).

What do you need to create a working setup?

• A PC (or SPARC server for Solaris) with 64bit PCI slots (at least one)
  • If you want he maximum performance than it's a good idea to have two separate PCI buses: One for the target mode HBA and one for the HBA where the drives are attached.
  • With SCST, IRQ sharing can be a problem on some mainbords (i386). Give the target mode HBA its own IRQ.
  • With SCST not every mainboard/HBA combination works flawlessly; i.e. QLA2200 can work without problems while QLA2342 won't...
• A supported FC HBA
  • best for SCST: Qlogic QLA 2xxx, I've tested QLA2200F and QLA2342 (yes, the QLA2200 works, even if it is not listed as supported...)
  • for Solaris an Emulex LP9000 series is the most inexpensive choice (caveat: doesn't work in FC-AL mode, fabric only)
• Storage hardware of your choice (SATA HDD's are IMHO the best option, but all HDD's supported by the OS will work)
  • If you want to use > 1Tb HDDs on Solaris you need to run on a 64bit platform since there is a 1024Gb limit in the 32 bit driver
• OS with target mode drivers available (see next point).
• Some luck ;-) (Or a good amount of it...)

For OSes with target mode drivers you have several options. I will list them in order of capabilities/stability:

• Linux with SCST drivers [1]
• Solaris/OpenSolaris (or whatever it is called now) COMSTAR [2]
• FreeBSD target mode drivers FreeBSD isp(4) manpage
• Linux with STGT drivers (iSCSI only at the moment, so not interesting)

It seems that there are also some 3rd-party drivers and apps for W*ndows, but they are rather expensive. So I can't say nothing at all if it works or not...

In this article I will focus on Linux with the SCST drivers.

I've tried the Solaris solution also and it seems to work quite nice. In a production environment this is probably the solution of choice since there is proper support and it seems more stable than SCST. However it doesn't come with the nice passthrough drivers that SCST has. There are some passthrough extensions for COMSTAR available but they seems to be experimental at best, but YMMV...

Setting up a working storage system with SCST

There are at least two ways to do this:

• Download the latest SCST sources from the homepage and try to patch the Linux kernel with them and compile the whole stuff

or

• Find a Linux distro that comes with SCST.

I tried both and found the compile-it-yourself option to be an unnecessary stony path to. It should be also mentioned that I never got something that worked in a stable manner... But you are encouraged to try it, since the author is a bit lousy when it comes to repairing broken device drivers.

Finding a Linux distro with SCST onboard was not that easy... In the end I found Openfiler [3] to be a good option. It is not only an FC target but can also serve your needs for an iSCSI target and NAS (Export with SMB, NFS, rsync and FTP). Since there is a fully fledged Linux under that nifty Web-based interface you can probably make it cooking your coffee... well, sort of...

Here is a list including the most important steps to get a working FC target:

• Download the latest OpenFiler ISO image from the aforementioned Site
• Assemble all your hardware, but leave out the FC HBA! This is very important. If you install OpenFiler with the FC HBA already plugged in chances are high that all will end with a complete mess...
• Install OpenFiler on your machine.
• Set up networking so that the machine has Internet access.
• Login as root on the system console. Run

 "conary update conary"

and

 "conary updateall".

• Reboot the system to verify that everything comes up well. Then shut down the system.
• Install the FC HBA.
• Start the system and verify that the FC HBA is recognized and properly initialized. You should see something like:

 [   12.425437] PCI: setting IRQ 10 as level-triggered
 [   12.425451] qla2xxx 0000:00:12.0: found PCI INT A -> IRQ 10
 [   12.488923] qla2xxx 0000:00:12.0: Found an ISP2200, irq 10, iobase 0xe08fe000  
 [   12.489921] qla2xxx 0000:00:12.0: Configuring PCI space...
 [   12.490131] qla2xxx 0000:00:12.0: Configure NVRAM parameters...
 [   12.582383] qla2xxx 0000:00:12.0: Verifying loaded RISC code...
 [   12.933393] qla2xxx 0000:00:12.0: firmware: requesting ql2200_fw.bin
 [   24.525110] IRQ 10/qla2xxx: IRQF_DISABLED is not guaranteed on shared IRQs
 [   24.525163] scsi2 : qla2xxx
 [   24.535729] qla2xxx 0000:00:12.0: 
 [   24.535735]  QLogic Fibre Channel HBA Driver: 8.02.01-k4
 [   24.535741]   QLogic QLA22xx - 
 [   24.535745]   ISP2200: PCI (33 MHz) @ 0000:00:12.0 hdma-, host#=2, fw=2.02.08

and some lines later

 [   46.697597] qla2xxx 0000:00:12.0: Cable is unplugged...

If you have some other fibre channel targets at your hand (i.e. disks) you can now plug them in and see if they are detected. If you can access these disks you are fine. Unplug them again and reboot. Now we can setup target mode support. To get a working target we have to set up two parts:

• The HBA target mode driver (qla2x00tgt)
• The high level software (SCST)

Lets start with the latter to confirm if it works. Due to some kernel patches there are some hickups on certain hardware configurations. If SCST doesn't work we can forget setting up something usefull. In this (rare) case, a different solution has to be found (no recipe yet, sorry).

To bring up SCST we do

 [root@bathor ~]# chkconfig scst on

We have to look up the WWN of the FC HBAs port we want to use as a target:

 [root@bathor ~]# cat /sys/class/fc_host/hostn/port_name
 0x210000e08b05194e

where n of hostn in the path is the number of the SCSI HBA (see dmesg output). It is important to look up the WWN before starting target mode and SCST because the directory and file structure in /sys changes as soon as SCST is running. When we are sure that we have identified the correct port and the belonging WWN we can start SCST:

 [root@bathor ~]# service scst start
 Loading and configuring the mid-level SCSI target SCST.

Note: We are not connected to a SAN yet... hopefully. Because when things go wrong here the chance is high that you mess everything up in your SAN!

Let's have a look at the dmesg output. We should see something like:

 [ 2236.597975] scst: Processing thread started, PID 2666
 [ 2236.599379] scst: Init thread started, PID 2667
 [ 2236.599437] scst: Task management thread started, PID 2668
 [ 2236.606960] scst: SCST version 2.0.0-pre1-procfs loaded successfully (max mem for commands 125MB, per device 50MB)
 [ 2236.606980] scst: Enabled features: TRACING
 [ 2236.621319] scst: Management thread started, PID 2669
 [ 2236.761905] scst: Virtual device handler vdisk_fileio for type 0 registered successfully
 [ 2236.761937] scst: Virtual device handler vdisk_blockio for type 0 registered successfully
 [ 2236.761953] scst: Virtual device handler vdisk_nullio for type 0 registered successfully
 [ 2236.761982] scst: Virtual device handler vcdrom for type 5 registered successfully

Everything fine so far...

On newer versions of OpenFiler, the HBA target mode is not enabled together with SCST. So we have to issue a

 [root@bathor ~]# chkconfig qla2x00tgt on

Back to SCST. For the most part, SCST can be managed through the scstadmin utility. We use it to activate target mode on the WWN we looked up previously:

 [root@bathor ~]# scstadmin -enable 21:00:00:e0:8b:05:19:4e
 Collecting current configuration: done.

         -> Enabling target mode for SCST host '0x210000e08b05194e'.

 All done.

Now it's time for a reboot. After the machine is up again we check the dmesg output (again). We hopefully find something like:

 [   44.132348] Initializing QLogic Fibre Channel HBA Driver target mode addon version 2.0.0
 [   44.139285] scst: Target 21:00:00:e0:8b:05:19:4e (rel ID 1) for template qla2x00tgt registered successfully
 [   44.139316] Target mode driver for QLogic 2x00 controller registered successfully
 [   44.139328] scst: Target template qla2x00tgt registered successfully

Nice! All the low- and midlevel stuff is running (it seems to) so we try to add a volume in the next step.

Todo:

• Create a volume for FC export
• Export that volume

Step 1: Use the Openfiler WebGUI and add an iSCSI volume (please RTFM for this)

Step 2:

• Locate the volume file. Mine is under /dev/1500g-vol-gr/fc_scst_250g_1
• Issue the following commands:

 scstadmin -enable 21:00:00:e0:8b:05:19:4e
 scstadmin -adddev 250G1 -handler vdisk -path /dev/1500g-vol-gr/fc_scst_250g_1 -options WRITE_THROUGH
 scstadmin -assigndev 250G1 -group Default -lun 0
 scstadmin -writeconfig /etc/scst.conf

The last command makes everything reboot proof. Please adjust the command parameters accordingly... (only examples here)

dmesg should tell us:

 [10329.248381] dev_vdisk: Attached SCSI target virtual disk 250G1 (file="/dev/1500g-vol-gr/fc_scst_250g_1"
 , fs=250016MB, bs=512, nblocks=512032768, cyln=250016)
 [10329.248457] scst: Attached to virtual device 250G1 (id 1)
                scst: Added device 250G1 to group Default (LUN 0, rd_only 0)

At this point a single initiator should be connected to the target mode host. The newly created volume should appear as a HDD on the initator. Try to make a file system, write and read some files...

When the target mode host is now connected to a SAN be shure that it can't see other active targets. On loops it means that no other targets can be active; on fabrics one can set up the zoning in way that other targets are isolated from the target mode host.

The reason behind this: The iniator driver is also active in the background and catches up every other target on the SAN. Unfortunately this happens in way which spoils the target mode driver and the system got hung up...

IRIX as an initiator

The target is in place now, so let us look at the initiator side. Under normal circumstances no special adjustments must be made. The virtual disks which are exported by SCST should behave like every other SCSI disk. Loop attached disk appear as a device icon on the desktop; fabric attached ones under their respective WWNs in ther /dev/disk directory.

However there are some limits in usability depending on the actual HBA being used. My experiences are as follows:

• QLA2200: No problems. Works flawlessly in FC-AL, FC-SW
• QLA24xx: see above.
• Adaptec based cards (dual XIO, PCI): Sorry, no way. FC-AL only, which doesn't work (N_port login failure).

Adaptec FC HBAs seem to be picky in general: I have one in my SGI VW320 which works with SCST, but there are also random login failures. Sometimes it works, sometimes not...

Not tested yet (If you have, feel free to add it here)

• LSI FC HBA (guess: will work)
• Prisa NetFX (guess: will not work, but chances are higher than the Adaptec ones (FC-SW support))

(to be continued...)

Small Computer System Interface

Host Bus Adapter

Scalable Processor Architecture

Peripheral Component Interconnect

Network File System

File Transfer Protocol

Non-Volatile Random Access Memory

Reduced Instruction Set Computer