This is a very quick HOWTO intended to provide a quick reference in how to setup and configure iSCSI-SCST. For a deeper info please refer to SCST and iSCSI-SCST README files. Installing and using iSCSI-SCST with scstadmin ============================================== 1. Download, build and install iSCSI-SCST. Start with downloading the iSCSI-SCST source code. You can either download the released version from the following URL: http://scst.sourceforge.net/downloads.html or you can download the latest development version by running the following command in a shell: svn co https://scst.svn.sourceforge.net/svnroot/scst/trunk scst While SCST and iSCSI-SCST work fine with an unpatched Linux kernel with the same or better speed as other open source iSCSI targets, if you want even better performance you need to patch and rebuild the kernel. Select the patch according to your kernel version (2.6.38.x in the example below): cd /usr/src/kernels/linux-2.6.38.8 patch -p1 < $HOME/scst/iscsi-scst/kernel/patches/put_page_callback-2.6.38.patch make clean If you can't find put_page_callback for your kernel, then the patching is not needed for it, so you can skip this and next (kernel rebuild and reboot) steps. Next, build and install the kernel: make && make modules make modules_install && make install For some Linux distributions (e.g. Ubuntu) the above steps do not rebuild the initial RAM disk. Make sure you regenerate the RAM disk before rebooting. Please look up the documentation of the update-initrd, update-initramfs or the mkinitrd command supplied with your distribution. Now reboot your system such that the newly built kernel is loaded. Once the system is again up and running the next step is to build and install SCST, iSCSI-SCST and scstadmin. With most distros (including RHEL 6) you can do that by running the following command from the SCST directory: make 2release make scst scst_install iscsi iscsi_install scstadm scstadm_install When using RHEL 5 or CentOS 5, use these commands instead: make 2release make enable_proc make scst scst_install iscsi iscsi_install scstadm scstadm_install iSCSI-SCST includes the following components: * kernel modules: iscsi-scst.ko scst.ko scst_vdisk.ko * iscsi-scstd - daemon * man pages * Obsolete configuration files located under iscsi-scst/etc/obsolete, need to be copied to /etc if you want to use them. In iSCSI-SCST 2.0 it is recommended to use scstadmin and SCST sysfs interface for that: * initiators.allow - used for assigning specific initiators to targets * initiators.deny - used for assigning specific initiators to targets * iscsi-scstd.conf - list of targets and their properties 2. Set up /etc/scst.conf The scst.conf configuration file is loaded at system boot time by the shell script /etc/init.d/scst via scstadmin. The target, device and LUN configuration is defined in this file. Generally, SCST and its components, including iSCSI-SCST, are kernel modules, hence they are using kernel interface for configuring, sysfs. However, using sysfs interface directly is cumbersome. Scstadmin utility allows to use regular plain text interface instead. You can load a config file into SCST sysfs using "scstadmin -config" and save currently running configuration into scst.conf using "scstadmin -write_config". See below the second section, if you want to use SCST sysfs interface directly. Here is the sample config file. Copy it in /etc/scst.conf HANDLER vdisk_fileio { DEVICE disk01 { filename /dev/ram0 nv_cache 1 } DEVICE disk02 { filename /dev/ram1 nv_cache 1 } } TARGET_DRIVER iscsi { enabled 1 TARGET iqn.2006-10.net.vlnb:tgt { LUN 0 disk01 LUN 1 disk02 enabled 1 } } Note that for each initiator LUN 0 must exist, hence you can't create targets and initiator groups without LUN 0. This is required by the SCSI protocol and is also documented in SCST's README file. Then to start scst and iscsi-scst services run the following command: /etc/init.d/scst start You should have the working iSCSI-SCST now. If at any later time you need to change scst.conf and reload it, you can run scstadmin directly: scstadmin -config /etc/scst.conf or restart SCST: /etc/init.d/scst restart Do the following steps if you need to automatically configure SCST upon reboot. 3. Create soft links in /etc/init.d This will make SCST to start automatically upon system boot. Use either chkconfig or update-rc.d, depending on the Linux distribution you are using. An example for Debian systems such as Ubuntu: update-rc.d scst defaults On openSUSE you can let SCST start automatically through the following commands: insserv scst 4. Restart SCST and iSCSI-SCST If you don't want to reboot now, you can make the above settings effective via the following commands: /etc/init.d/scst restart Installing and using iSCSI-SCST without using scstadmin via /sys interface ========================================================================== This is advanced section you should not need for basic services. Scstadmin should be fully sufficient for you. Use those steps only if you need to do something too advanced for scstadmin. For instance, keep your configuration is a database instead of plain file. In an iSCSI-SCST system, three components interact to export your devices and file systems to iSCSI initiators on remote hosts: Targets, LUNS, and Devices. A Target represents the host and acts as a container for LUNs. A LUN (Logical Unit Number) represents the address of a Device and provides access control for initiators, as well as acting as a container for Devices. A Device represents the actual block device or file system you wish to make available to initiators. You have to create targets first, then create LUNs, then assign LUNs to targets, then define devices and assign them to a LUN. Finally, you have to enable your Targets and the SCST system. Once the steps (below) have been completed, you should be able to find and access your exported devices from authorized remote initiators. First repeat step 1 the first section but leave out "scstadm scstadm_install" from the make command in step 1. Next load the required kernel modules as follows: modprobe scst_disk modprobe scst_vdisk modprobe iscsi-scst Module scst_disk allows to export local SCSI (SAS, etc.) devices in pass-through mode, scst_vdisk allows to create virtual LUNs based on files and block devices, module iscsi-scst is the main iSCSI-SCST module. Then run iSCSI-SCST user space service: iscsi-scstd Creating targets ---------------- echo "add_target iqn.2007-05.com.example:storage.iscsi-scst-1" > /sys/kernel/scst_tgt/targets/iscsi/mgmt This will add target iqn.2007-05.com.example:storage.iscsi-scst-1. Changing targets' parameters ---------------------------- echo "8192" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example\:storage.iscsi-scst-1/MaxRecvDataSegmentLength This will change MaxRecvDataSegmentLength parameter of target iqn.2007-05.com.example:storage.iscsi-scst-1 to 8192. You can read it then by: # cat /sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example\:storage.iscsi-scst-1/MaxRecvDataSegmentLength 8192 [key] The mark "[key]" shows high level management tools, like scstadmin, that this attribute has not default value. You can ignore it. Add vdisk devices ----------------- dd if=/dev/zero of=/disk1 bs=1M count=1024 This will create a backend file /disk1 for our future virtual device disk1. echo "add_device disk1 filename=/disk1; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt This will add new FILEIO device disk1 with backend file /disk1 and NV_CACHE option enabled. For a block device usually it is more efficient to use BLOCKIO handler: echo "add_device dev1 filename=/dev/dev1" >/sys/kernel/scst_tgt/handlers/vdisk_blockio/mgmt Listing currently registered SCST devices ----------------------------------------- You can see the list of available devices: # ls -1 /sys/kernel/scst_tgt/devices 0:0:0:0 1:0:0:0 3:0:0:0 3:0:0:1 3:0:0:2 3:0:0:3 3:0:0:4 3:0:0:5 3:0:0:6 3:0:0:7 2:0:0:0 4:0:0:0 4:0:0:5 4:0:0:6 4:0:0:7 4:0:0:8 4:0:0:9 4:0:1:0 4:0:1:1 4:0:1:2 4:0:1:3 4:0:1:4 5:0:0:0 5:0:0:5 5:0:0:6 5:0:0:7 5:0:0:8 5:0:0:9 5:0:1:0 5:0:1:1 5:0:1:2 5:0:1:3 5:0:1:4 disk1 Defining LUN masking -------------------- LUN masking defines which LUNs which initiator can see. In order to associate specific LUNs with target iqn.2007-05.com.example:storage.iscsi-scst-1, do the following: echo "add 4:0:0:8 0" > /sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example\:storage.iscsi-scst-1/luns/mgmt This will assign the LUN 0 for pass-through device 4:0:0:8 to the target iqn.2007-05.com.example:storage.iscsi-scst-1. Note, you must have LUN 0 for each group. That's a SCSI requirement (documented in SCST's README). echo "add disk1 1 read_only=1" > /sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example\:storage.iscsi-scst-1/luns/mgmt This will assign the read only LUN 1 for virtual device disk1 to the target iqn.2007-05.com.example:storage.iscsi-scst-1. Deleting a LUN from a group --------------------------- Run the following command: echo "del X" > /sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example\:storage.iscsi-scst-1/luns/mgmt This will remove the LUN X from the target iqn.2007-05.com.example:storage.iscsi-scst-1. For more information about LUN masking, refer to SCST README, section "Access and devices visibility management (LUN masking)". Enabling targets ---------------- After you created and configured target, you should enable it: echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example\:storage.iscsi-scst-1/enabled Then enable the iSCSI-SCST driver: echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled Assigning targets to specific initiators ---------------------------------------- In order to assign targets to specific initiators, you should use LUN masking defined above. If an initiator assigned to an empty security group of a target, the initiator will not see this target in the SendTargets discovery and will not be able to connect to it. Additionally, you can make the target be visible only via selected portals using allowed_portal sysfs attribute. Read iSCSI-SCST's README file for more details about this facility. Installing and using iSCSI-SCST without using scstadmin via /proc interface =========================================================================== First repeat steps 1 and 2 from the previous section but leave out "scstadm scstadm_install" from the make command in step 1. Next set up /etc/iscsi-scstd.conf You can do this by e.g. running the following shell commands: echo "Target $(sed -n 's/InitiatorName=//p' /etc/iscsi/initiatorname.iscsi):storage" >/etc/iscsi-scstd.conf cat /etc/iscsi-scstd.conf Next load the scst_disk kernel module as follows: modprobe scst_disk The shell script /etc/init.d/iscsi-scst can now be used to start/stop/restart or check the status of iSCSI-SCST. Selecting devices to be used by SCST ------------------------------------ You can see the list of available devices: # cat /proc/scsi_tgt/scsi_tgt Device (host:ch:id:lun or name) Device handler 0:0:0:0 dev_disk 1:0:0:0 dev_disk 3:0:0:0 dev_disk 3:0:0:1 dev_disk 3:0:0:2 dev_disk 3:0:0:3 dev_disk 3:0:0:4 dev_disk 3:0:0:5 dev_disk 3:0:0:6 dev_disk 3:0:0:7 dev_disk 2:0:0:0 none 4:0:0:0 none 4:0:0:5 dev_disk 4:0:0:6 dev_disk 4:0:0:7 dev_disk 4:0:0:8 dev_disk 4:0:0:9 dev_disk 4:0:1:0 dev_disk 4:0:1:1 dev_disk 4:0:1:2 dev_disk 4:0:1:3 dev_disk 4:0:1:4 dev_disk 5:0:0:0 none 5:0:0:5 dev_disk 5:0:0:6 dev_disk 5:0:0:7 dev_disk 5:0:0:8 dev_disk 5:0:0:9 dev_disk 5:0:1:0 dev_disk 5:0:1:1 dev_disk 5:0:1:2 dev_disk 5:0:1:3 dev_disk 5:0:1:4 dev_disk LUN masking ----------- SCST defines security groups. For each group, you can add LUNs. If you want to have all LUNs available for all targets, just add them to the "Default" group: # echo "add 5:0:0:8 0" >/proc/scsi_tgt/groups/Default/devices # cat /proc/scsi_tgt/groups/Default/devices Device (host:ch:id:lun or name) Virtual lun Options 5:0:0:8 0 Now, the LUN "5:0:0:8" was added to the "Default" group as LUN #0. Note, that you must have LUN 0. That's a SCSI requirement (documented in SCST's README). Defining LUN masking -------------------- In order to associate specific LUNs with specific targets, do the following: * Create a group for the target: echo "add_group Default_iqn.2007-05.com.example:storage.iscsi-scst-1" >/proc/scsi_tgt/scsi_tgt * Add LUNs to the group: echo "add 4:0:0:8 0" > /proc/scsi_tgt/groups/Default_iqn.2007-05.com.example\:storage.iscsi-scst-1/devices This will assign the LUN "4:0:0:8" to the target iqn.2007-05.com.example:storage.iscsi-scst-1. Again, you must have LUN 0 for each group. Deleting a LUN from a group --------------------------- Run the following command: echo "del 4:0:0:8 0" > /proc/scsi_tgt/groups/Default_iqn.2007-05.com.example\:storage.iscsi-scst-1/devices This will remove the LUN "4:0:0:8" from the target iqn.2007-05.com.example:storage.iscsi-scst-1. Deleting a group ---------------- Run the following command: echo "del_group Default_iqn.2007-05.com.example:storage.iscsi-scst-1" >/proc/scsi_tgt/scsi_tgt For more information about LUN masking, refer to SCST README, section "Access and devices visibility management (LUN masking)". Creating targets using iscsi-scstd.conf --------------------------------------- The easiest way to create targets is to define them in /etc/iscsi-scstd.conf. An example can be found in etc/iscsi-scstd.conf. You need to have this file under /etc/ before starting iSCSI-SCST. Assigning targets to specific initiators ---------------------------------------- In order to assign targets to specific initiators, you need to have /etc/initiators.allow and /etc/initiators.deny. You can find example files in etc/initiators.allow and etc/initiators.deny. Note that all targets are allowed to all initiators by default, so if you want to use /etc/initiators.allow, you will need to have /etc/initiators.deny that looks like this: ALL ALL This will deny all initiators expect for those defined in /etc/initiators.allow. Useful examples of iscsi-scst-adm: Show a specific target: # iscsi-scst-adm --op show --tid=1 QueuedCommands=0 Show the configured parameters for a specific target: # iscsi-scst-adm --op show --tid=1 --sid=0 InitialR2T=No ImmediateData=Yes MaxConnections=1 MaxRecvDataSegmentLength=2097152 MaxXmitDataSegmentLength=131072 MaxBurstLength=2097152 FirstBurstLength=262144 DefaultTime2Wait=2 DefaultTime2Retain=0 MaxOutstandingR2T=1 DataPDUInOrder=Yes DataSequenceInOrder=Yes ErrorRecoveryLevel=0 HeaderDigest=None DataDigest=None OFMarker=No IFMarker=No OFMarkInt=Reject IFMarkInt=Reject Bart Van Assche Erez Zilber Vladislav Bolkhovitin