SCST technical description

Vladislav Bolkhovitin

Version 3.0.0 for SCST 3.0.0 and later

Appendix

1. Introduction

SCST is a SCSI target mid-level subsystem for Linux. It provides unified consistent interface between SCSI target drivers, backend device handlers and Linux kernel as well as simplifies target drivers development as much as possible.

It has the following features:

2. Terms and Definitions

SCSI initiator device

A SCSI device that originates service and task management requests to be processed by a SCSI target device and receives device service and task management responses from SCSI target devices.

SCSI target device

A SCSI device that receives device service and task management requests for processing and sends device service and task management responses to SCSI initiator devices or drivers.

SCST session

SCST session is the object that describes relationship between a remote initiator and SCST via a target driver. All the commands from the remote initiator is passed to SCST in the session. For example, for connection oriented protocols, like iSCSI, SCST session could be mapped to TCP connection (as well as iSCSI session). SCST session is equivalent of SCSI I_T nexus object.

Local SCSI initiator

A SCSI initiator that is located on the same host as SCST subsystem. Examples are sg and st drivers.

Remote SCSI initiator

A SCSI initiator that is located on the remote host for SCST subsystem and makes client connections to SCST via SCST target drivers.

SCSI target driver

A Linux hardware or logical driver that acts as a SCSI target for remote SCSI initiators, i.e. accepts remote connections, passes incoming SCSI requests to SCST and sends SCSI responses from SCST back to their originators.

Device (backend) handler driver

Also known as "device type specific driver" or "dev handler", SCST driver, which helps SCST to analyze incoming requests and determine parameters, specific to various types of devices as well as perform some processing. See below for more details.

3. SCST Core Architecture

SCST accepts commands and passes them to SCSI mid-level at the same way as SCSI high-level drivers (sg, sd, st) do. Figure 1 shows interaction between SCST, its drivers and Linux SCSI subsystem.


Interaction between SCST, its drivers and Linux SCSI subsystem.

4. Target drivers

4.1 struct scst_tgt_template

To work with SCST a target driver must register its template in SCST by calling scst_register_target_template(). The template lets SCST know the target driver's entry points. It is defined as the following:

struct scst_tgt_template 
{
        int sg_tablesize;
        const char name[SCST_MAX_NAME];

        unsigned unchecked_isa_dma:1;
        unsigned use_clustering:1;
        unsigned no_clustering:1;

        unsigned xmit_response_atomic:1; 
        unsigned rdy_to_xfer_atomic:1;

        unsigned no_proc_entry:1;

        int max_hw_pending_time;

        int threads_num;

        int (*detect) (struct scst_tgt_template *tgt_template);
        int (*release)(struct scst_tgt *tgt);

        int (*xmit_response)(struct scst_cmd *cmd);
        int (* rdy_to_xfer)(struct scst_cmd *cmd);

        void (*on_hw_pending_cmd_timeout) (struct scst_cmd *cmd);

        void (*on_free_cmd) (struct scst_cmd *cmd);

        int (*alloc_data_buf) (struct scst_cmd *cmd);

        void (*preprocessing_done) (struct scst_cmd *cmd);

        int (*pre_exec) (struct scst_cmd *cmd);

        void (*task_mgmt_affected_cmds_done) (struct scst_mgmt_cmd *mgmt_cmd);
        void (*task_mgmt_fn_done)(struct scst_mgmt_cmd *mgmt_cmd);

        int (*report_aen) (struct scst_aen *aen);

        int (*read_proc) (struct seq_file *seq, struct scst_tgt *tgt);
        int (*write_proc) (char *buffer, char **start, off_t offset,
                int length, int *eof, struct scst_tgt *tgt);

        int (*get_initiator_port_transport_id) (struct scst_session *sess,
                uint8_t **transport_id);
}

Where:

Functions xmit_response(), rdy_to_xfer() are expected to be non-blocking, i.e. return immediately and don't wait for actual data transfer to finish. Blocking in such command could negatively impact on overall system performance. If blocking is necessary, it is worth to consider creating dedicated thread(s) in target driver, to which the commands would be passed and which would perform blocking operations instead of SCST. If the function allowed to sleep or not is defined by "atomic" attribute of the cmd that can be get via scst_cmd_atomic(), which is true, if sleeping is not allowed. In this case, if the function requires sleeping, it can return SCST_TGT_RES_NEED_THREAD_CTX in order to be recalled in the thread context, where sleeping is allowed.

Functions task_mgmt_fn_done() and report_aen() are recommended to be non-blocking as well. Blocking there will stop all management processing for all target drivers in the system (there is only one management thread in the system).

Functions xmit_response() and rdy_to_xfer() can return the following error codes:

More about xmit_response()

As already written above, function xmit_response() should transmit the response data and the status from the cmd parameter.

Sense data, if any, is contained in the buffer, returned by scst_cmd_get_sense_buffer(), with length, returned by scst_cmd_get_sense_buffer_len(). SCST always works in autosense mode. If a low-level SCSI driver/device doesn't support autosense mode, SCST will issue REQUEST SENSE command, if necessary. Thus, if CHECK CONDITION established, target driver will always see sense in the sense buffer and isn't required to request the sense manually.

After the response is completely sent, the target should call scst_tgt_cmd_done() function in order to allow SCST to free the command.

Function xmit_response() returns one of the SCST_TGT_RES_* constants, described above. Pay attention to "atomic" attribute of the cmd, which can be get via scst_cmd_atomic(): it is true if the function called in the atomic (non-sleeping) context.

To detect aborted commands xmit_response() must in the beginning check return status of function scst_cmd_aborted_on_xmit(). If it's true, xmit_response() must call scst_set_delivery_status(cmd, SCST_CMD_DELIVERY_ABORTED) and terminate further processing by calling scst_tgt_cmd_done(cmd, SCST_CONTEXT_SAME).

4.2 Target driver registration functions

scst_register_target_template()

Function scst_register_target_template() is defined as the following:

int scst_register_target_template(
        struct scst_tgt_template *vtt)

Where:

Returns 0 on success or appropriate error code otherwise.

scst_register_target()

Function scst_register_target() is defined as the following:

struct scst_tgt *scst_register_target(
        struct scst_tgt_template *vtt)

Where:

Returns target structure based on template vtt or NULL in case of error.

4.3 Target driver unregistration functions

In order to unregister itself target driver should at first call scst_unregister_target() for all its adapters and then call scst_unregister_target_template() for its template.

scst_unregister_target()

Function scst_unregister_target() is defined as the following:

void scst_unregister_target(
        struct scst_tgt *tgt)

Where:

scst_unregister_target_template()

Function scst_unregister_target_template() is defined as the following:

void scst_unregister_target_template(
        struct scst_tgt_template *vtt)

Where:

5. Device specific drivers (backend device handlers)

Device specific drivers are add-ons for SCST, which help SCST to analyze incoming requests and determine parameters, specific to various types of devices as well as actually execute specified SCSI commands. Device handlers are intended for the following:

Device handlers considered to be part of SCST, so they could directly access any fields in SCST's structures as well as use the corresponding functions.

Without appropriate device handler SCST hides devices of this type from remote initiators and returns HARDWARE ERROR sense data to any requests to them.

5.1 Structure scst_dev_type

Structure scst_dev_type is defined as the following:

struct scst_dev_type
{
        char name[];
        int type;

        unsigned parse_atomic:1;
        unsigned alloc_data_buf_atomic:1;
        unsigned dev_done_atomic:1;

        unsigned no_proc:1;

        unsigned exec_sync:1;

        unsigned pr_cmds_notifications:1;

        int threads_num;
        enum scst_dev_type_threads_pool_type threads_pool_type;

        int (*attach) (struct scst_device *dev);
        void (*detach) (struct scst_device *dev);

        int (*attach_tgt) (struct scst_tgt_device *tgt_dev);
        void (*detach_tgt) (struct scst_tgt_device *tgt_dev);

        int (*parse) (struct scst_cmd *cmd);
        int (*alloc_data_buf) (struct scst_cmd *cmd);
        int (*exec) (struct scst_cmd *cmd);
        int (*dev_done) (struct scst_cmd *cmd);
        int (*on_free_cmd) (struct scst_cmd *cmd);

        int (*task_mgmt_fn) (struct scst_mgmt_cmd *mgmt_cmd, 
                struct scst_tgt_dev *tgt_dev);

        int (*read_proc) (struct seq_file *seq, struct scst_dev_type *dev_type);
        int (*write_proc) (char *buffer, char **start, off_t offset,
                int length, int *eof, struct scst_dev_type *dev_type);
}

Where:

5.2 Device specific drivers registration

scst_register_dev_driver()

To work with SCST a device specific driver must register itself in SCST by calling scst_register_dev_driver(). It is defined as the following:

int scst_register_dev_driver(
        struct scst_dev_type *dev_type)

Where:

The function returns 0 on success or appropriate error code otherwise.

scst_register_virtual_device()

To create a virtual device a device handler must register it in SCST by calling scst_register_virtual_device(). It is defined as the following:

int scst_register_virtual_device(
        struct scst_dev_type *dev_handler,
        const char *dev_name)

Where:

The function returns ID assigned to the device on success, or negative value otherwise.

All local real SCSI devices will be registered and unregistered by the SCST core automatically, so pass-through dev handlers don't have to worry about it.

5.3 Device specific drivers unregistration

scst_unregister_virtual_device()

Virtual devices unregistered by calling scst_unregister_virtual_device(). It is defined as the following:

void scst_unregister_virtual_device(
        int id)

Where:

scst_unregister_dev_driver()

Device specific driver is unregistered by calling scst_unregister_dev_driver(). It is defined as the following:

void scst_unregister_dev_driver(
        struct scst_dev_type *dev_type)

Where:

6. SCST sessions

6.1 SCST sessions registration

When target driver determines that it needs to create new SCST session (for example, by receiving new TCP connection), it should call scst_register_session(), that is defined as the following:

struct scst_session *scst_register_session(
        struct scst_tgt *tgt,
        int atomic,
        const char *initiator_name,
        void *tgt_priv,
        void *result_fn_data,
        void (*result_fn) (
                struct scst_session *sess,
                void *data,
                int result))

Where:

A session creation and initialization is a complex task, which requires sleeping state, so it can't be fully done in interrupt context. Therefore the "bottom half" of it, if scst_register_session() is called from atomic context, will be done in SCST thread context. In this case scst_register_session() will return not completely initialized session, but the target driver can supply commands to this session via scst_rx_cmd(). Those commands processing will be delayed inside SCST until the session initialization is finished, then their processing will be restarted. The target driver will be notified about finish of the session initialization by function result_fn(). On success the target driver could do nothing, but if the initialization fails, the target driver must ensure that no more new commands being sent or will be sent to SCST after result_fn() returns. All already sent to SCST commands for failed session will be returned in xmit_response() with BUSY status. In case of failure the driver shall call scst_unregister_session() inside result_fn(), it will NOT be called automatically.

Thus, scst_register_session() can be safely called from IRQ context.

6.2 SCST sessions unregistration

SCST session unregistration basically is the same, except that instead of atomic parameter there is wait one.

void scst_unregister_session(
        struct scst_session *sess,
        int wait,
        void (*unreg_done_fn)(
                struct scst_session *sess))

Where:

All outstanding commands will be finished regularly. After scst_unregister_session() returned no new commands must be sent to SCST via scst_rx_cmd(). Also, the caller must ensure that no scst_rx_cmd() or scst_rx_mgmt_fn_*() is called in parallel with scst_unregister_session().

Function scst_unregister_session()/ can be called before result_fn() of scst_register_session() called, i.e. during the session registration/initialization.

7. Commands processing and interaction between SCST core and its drivers

Consider simplified commands processing example. It assumes that target driver doesn't need own memory allocation, i.e. not defined alloc_data_buf() callback. Example of such target driver is qla2x00t.

The commands processing by SCST started when target driver calls scst_rx_cmd(). This function returns SCST's command. Then the target driver finishes the command's initialization, for example, storing necessary target driver specific data there, and calls scst_cmd_init_done() telling SCST that it can start the command processing. Then SCST translates the command's LUN to local device, determines the command's data direction and required data buffer size by calling appropriate device handler's parse() callback function. Then:

When the command is finished by SCSI mid-level, device handler's dev_done() callback is called to notify it about the command's completion. Then in order to send its response the target's xmit_response() callback is called. When the response, including data, if any, is transmitted, the target will call scst_tgt_cmd_done() to tell SCST that it can free the command and its data buffer.

Then during the command's deallocation device handler's and the target's on_free_cmd() callback will be called in this order, if set.

This sequence is illustrated on Figure 2. To simplify the picture, sign "..." means SCST's waiting state for the corresponding command to complete. During this state SCST and its drivers continue processing of other commands, if there are any. One way arrow, for example to xmit_response(), means that after this function returns, nothing valuable for the current command will be done and SCST goes sleeping or to the next command processing until the corresponding event happens.


The commands processing flow

7.1 The commands processing functions

scst_rx_cmd()

Function scst_rx_cmd() creates and sends new command to SCST. Returns the command on success or NULL otherwise. It is defined as the following:

struct scst_cmd *scst_rx_cmd(
        struct scst_session *sess,
        const uint8_t *lun,
        int lun_len,
        const uint8_t *cdb,
        int cdb_len,
        int atomic)

Where:

scst_cmd_init_done()

Function scst_cmd_init_done() notifies SCST that the driver finished its part of the command initialization, and the command is ready for execution. It is defined as the following:

void scst_cmd_init_done(
        struct scst_cmd *cmd,
        enum scst_exec_context pref_context)

Where:

scst_rx_data()

Function scst_rx_data() notifies SCST that the driver received all the necessary data and the command is ready for further processing. It is defined as the following:

void scst_rx_data(
        struct scst_cmd *cmd, 
        int status,
        enum scst_exec_context pref_context)

Where:

Parameter status can have one of the following values:

scst_tgt_cmd_done()

Function scst_tgt_cmd_done() notifies SCST that the driver has sent the data and/or response. It must not been called if there are an error and xmit_response() returned something other, than SCST_TGT_RES_SUCCESS. It is defined as the following:

void scst_tgt_cmd_done(
        struct scst_cmd *cmd,
        enum scst_exec_context pref_context)

Where:

7.2 The commands processing context

Execution context often is a major problem in the kernel drivers development, because many contexts, like IRQ context, greatly limit available functionality, therefore require additional complex code in order to pass processing to more simple context. SCST does its best to undertake most of the context handling.

On the initialization time SCST creates for internal command processing as many threads as there are processors in the system or specified by user via scst_threads module parameter. Similarly, as many tasklets created as there are processors in the system.

Each command can be processed in one of four contexts:

  1. Directly, i.e. in the caller's context, without limitations
  2. Directly atomically, i.e. with sleeping forbidden
  3. In the SCST's internal threads
  4. In the SCST's per processor tasklets

The target driver sets this context as pref_context parameter for SCST functions. Additionally, target's template's xmit_response_atomic and rdy_to_xfer_atomic flags have direct influence on the context. If one of them is false, the corresponding function will never be called in the atomic context and, if necessary, the command will be rescheduled to one of the SCST's threads.

SCST in some circumstances can change preferred context to less restrictive one, for example, for large data buffer allocation, if there is not enough GFP_ATOMIC memory.

Preferred context constants

There are the following preferred context constants:

7.3 SCST commands' processing states

There are the following processing states, which a SCST command passes through during execution and which could be returned by device handler's parse() and dev_done() (but not all states are allowed to be returned):

8. Task management functions

There are the following task management functions supported:

All task management functions return completion status via task_mgmt_fn_done() when the affected SCSI commands (tasks) are actually aborted, i.e. guaranteed never be executed any time later.

8.1 scst_rx_mgmt_fn_tag()

Function scst_rx_mgmt_fn_tag() tells SCST to perform the specified task management function, based on the command's tag. Can be used only for SCST_ABORT_TASK.

It is defined as the following:

int scst_rx_mgmt_fn_tag(
        struct scst_session *sess,
        int fn,
        uint32_t tag,
        int atomic,
        void *tgt_priv)

Where:

Returns 0 if the command was successfully created and scheduled for execution, error code otherwise. On success, the completion status of the command will be reported asynchronously via task_mgmt_fn_done() driver's callback.

8.2 scst_rx_mgmt_fn_lun()

Function scst_rx_mgmt_fn_lun() tells SCST to perform the specified task management function, based on the LUN. Currently it can be used for any function, except SCST_ABORT_TASK.

It is defined as the following:

int scst_rx_mgmt_fn_lun(
        struct scst_session *sess,
        int fn,
        const uint8_t *lun,
        int lun_len,
        int atomic,
        void *tgt_priv);

Where:

Returns 0 if the command was successfully created and scheduled for execution, error code otherwise. On success, the completion status of the command will be reported asynchronously via task_mgmt_fn_done() driver's callback.

Possible status constants which can be returned by scst_mgmt_cmd_get_status():

9. SGV cache

SCST SGV cache is a memory management subsystem in SCST. One can call it a "memory pool", but Linux kernel already have a mempool interface, which serves different purposes. SGV cache provides to SCST core, target drivers and backend dev handlers facilities to allocate, build and cache SG vectors for data buffers. The main advantage of it is the caching facility, when it doesn't free to the system each vector, which is not used anymore, but keeps it for a while (possibly indefinitely) to let it be reused by the next consecutive command. This allows to:

The freed SG vectors are kept by the SGV cache either for some (possibly indefinite) time, or, optionally, until the system needs more memory and asks to free some using the set_shrinker() interface. Also the SGV cache allows to:

9.1 Implementation

From implementation POV the SGV cache is a simple extension of the kmem cache. It can work in 2 modes:

  1. With fixed size buffers.
  2. With a set of power 2 size buffers. In this mode each SGV cache (struct sgv_pool) has SGV_POOL_ELEMENTS (11 currently) of kmem caches. Each of those kmem caches keeps SGV cache objects (struct sgv_pool_obj) corresponding to SG vectors with size of order X pages. For instance, request to allocate 4 pages will be served from kmem cache[2], since the order of the of number of requested pages is 2. If later request to allocate 11KB comes, the same SG vector with 4 pages will be reused (see below). This mode is in average allows less memory overhead comparing with the fixed size buffers mode.

Consider how the SGV cache works in the set of buffers mode. When a request to allocate new SG vector comes, sgv_pool_alloc() via sgv_get_obj() checks if there is already a cached vector with that order. If yes, then that vector will be reused and its length, if necessary, will be modified to match the requested size. In the above example request for 11KB buffer, 4 pages vector will be reused and modified using trans_tbl to contain 3 pages and the last entry will be modified to contain the requested length - 2*PAGE_SIZE. If there is no cached object, then a new sgv_pool_obj will be allocated from the corresponding kmem cache, chosen by the order of number of requested pages. Then that vector will be filled by pages and returned.

In the fixed size buffers mode the SGV cache works similarly, except that it always allocate buffer with the predefined fixed size. I.e. even for 4K request the whole buffer with predefined size, say, 1MB, will be used.

In both modes, if size of a request exceeds the maximum allowed for caching buffer size, the requested buffer will be allocated, but not cached.

Freed cached sgv_pool_obj objects are actually freed to the system either by the purge work, which is scheduled once in 60 seconds, or in sgv_shrink() called by system, when it's asking for memory.

9.2 Interface

sgv_pool *sgv_pool_create()

struct sgv_pool *sgv_pool_create(
        const char *name,
        enum sgv_clustering_types clustered, int single_alloc_pages,
        bool shared, int purge_interval)

This function creates and initializes an SGV cache. It has the following arguments:

Returns the resulting SGV cache or NULL in case of any error.

void sgv_pool_del()

void sgv_pool_del(
        struct sgv_pool *pool)

This function deletes the corresponding SGV cache. If the cache is shared, it will decrease its reference counter. If the reference counter reaches 0, the cache will be destroyed.

void sgv_pool_flush()

void sgv_pool_flush(
        struct sgv_pool *pool)

This function flushes, i.e. frees, all the cached entries in the SGV cache.

void sgv_pool_set_allocator()

void sgv_pool_set_allocator(
        struct sgv_pool *pool,
        struct page *(*alloc_pages_fn)(struct scatterlist *sg, gfp_t gfp, void *priv),
        void (*free_pages_fn)(struct scatterlist *sg, int sg_count, void *priv));

This function allows to set for the SGV cache a custom pages allocator. For instance, scst_user uses such function to supply to the cache mapped from user space pages.

alloc_pages_fn() has the following parameters:

This function should return the allocated page or NULL, if no page was allocated.

free_pages_fn() has the following parameters:

struct scatterlist *sgv_pool_alloc()

struct scatterlist *sgv_pool_alloc(
        struct sgv_pool *pool,
        unsigned int size,
        gfp_t gfp_mask,
        int flags,
        int *count,
        struct sgv_pool_obj **sgv,
        struct scst_mem_lim *mem_lim,
        void *priv)

This function allocates an SG vector from the SGV cache. It has the following parameters:

This function returns pointer to the resulting SG vector or NULL in case of any error.

void sgv_pool_free()

void sgv_pool_free(
        struct sgv_pool_obj *sgv,
        struct scst_mem_lim *mem_lim)

This function frees previously allocated SG vector, referenced by SGV cache object sgv.

void *sgv_get_priv(struct sgv_pool_obj *sgv)

void *sgv_get_priv(
        struct sgv_pool_obj *sgv)

This function allows to get the allocation private data for this SGV cache object sgv. The private data are set by sgv_pool_alloc().

void scst_init_mem_lim()

void scst_init_mem_lim(
        struct scst_mem_lim *mem_lim)

This function initializes memory limits structure mem_lim according to the current system configuration. This structure should be latter used to track and limit allocated by one or more SGV caches memory.

9.3 Runtime information and statistics.

SGV cache runtime information and statistics is available in /proc/scsi_tgt/sgv.

10. Target driver qla2x00t

Target driver qla2x00t allows to use QLogic 2xxx based adapters in the target (server) mode.

It consists from two parts:

The initiator driver qla2xxx was changed to:

The changes are relatively small (few thousands lines big patch) and local.

The changed qla2xxx is still capable to work as initiator only. Mode, when a host acts as initiator and target simultaneously, is supported as well.

Since firmware interface for 24xx+ chips is fundamentally different from earlier versions, qla2x00t generally contains 2 separate drivers sharing some common processing.

10.1 Driver initialization

On initialization, qla2x00tgt registers its SCST template tgt2x_template in the SCST core. Then during template registration SCST core calls detect() callback which is function q2t_target_detect().

In this function qla2x00tgt registers its callbacks in qla2xxx by calling qla2xxx_tgt_register_driver(). Qla2xxx_tgt_register_driver() stores pointer to the being registered callbacks in variable qla_target.

Then q2t_target_detect() calls qla2xxx_add_targets(), which calls for each known local FC port (HBA instance) qla_target.tgt_host_action() callback with ADD_TARGET action. Then q2t_host_action() calls q2t_add_target() which registers SCST target for this FC port.

If later a new FC port is hot added, qla2x00_probe_one() will also call for all new local ports qla_target.tgt_host_action() with ADD_TARGET action.

10.2 Driver unload

When a local FC port is being removed, the Linux kernel calls qla2x00_remove_one(), which then qla_target.tgt_host_action() with REMOVE_TARGET action.

Then q2t_host_action() calls q2t_remove_target(), which unregisters the corresponding SCST target in SCST. During unregistration SCST core calls release() callback of tgt2x_template, which is q2t_target_release().

Then q2t_target_release() calls q2t_target_stop(). Then q2t_target_stop() marks this target as stopped by setting flag tgt_stop. When this flag is set, all incoming from initiators commands are refused.

Then q2t_target_stop() schedules deletion of all sessions of the target.

Then q2t_target_stop() waits until all outstanding commands finished and sessions deleted.

Then q2t_target_stop(), if necessary, calls qla2x00_disable_tgt_mode() to disables target mode, which disables target mode of the corresponding HBA and resets it. Then qla2x00_disable_tgt_mode() waits until reset finished.

Then q2t_target_stop() returns and then q2t_target_release() frees the target.

If module qla2x00tgt is being unloaded, q2t_exit() at first takes q2t_unreg_rwsem on writing. Taking it is necessary to make sure that q2t_host_action() will not be active during qla2x00tgt unload.

Then q2t_exit() calls scst_unregister_target_template() for tgt2x_template, which then in a loop will unregister all QLA SCST targets from SCST as described above.

10.3 Enabling target mode

When command to enable target mode received, qla_target.tgt_host_action() with action ENABLE_TARGET_MODE called. Then q2t_host_action() goes over all discovered remote of the being enabled target and adds SCST sessions for all them.

Then it calls qla2x00_enable_tgt_mode(), which enables target mode of the corresponding HBA and resets it. Then qla2x00_enable_tgt_mode() waits until reset finished.

During reset firmware initialization functions detect that target mode is enables and initialize the firmware accordingly.

10.4 Disabling target mode

When command to disable target mode received, qla_target.tgt_host_action() with action DISABLE_TARGET_MODE called. Then q2t_host_action() calls q2t_target_stop(), which processes as describe above.

10.5 SCST sessions management

As required by SCSI and FC standards, each remote initiator FC port has the corresponding SCST session.

Since qla2xxx is not intended to strictly maintain database of remote initiator FC ports as it is needed for target mode, qla2x00t uses mixed approach for SCST sessions management, when both qla2xxx and QLogic firmware generate events and information about currently active remote FC ports.

Remote FC ports management also has to handle changing FC and loop IDs after fabric events, so it needs to constantly monitor FC and loop IDs of the registered FC ports. This is implemented by checks in q2t_create_sess() that being registered FC port already has SCST session and q2t_check_fcport_exist() in q2t_del_sess_work_fn(). See below for more info.

Interaction with qla2xxx is implemented using tgt_fc_port_added() and tgt_fc_port_deleted() qla_target's callbacks.

Callback tgt_fc_port_added() called by qla2xxx when the target driver detects new remote FC port. Assigned to it q2t_fc_port_added() checks if an SCST session already exists for this remote FC port and, if not, creates it.

Callback tgt_fc_port_deleted() called by qla2xxx when it deletes a remote FC port from its database. Assigned to it q2t_fc_port_deleted() checks if an SCST session already exists for this remote FC port and, if yes, schedules it for deletion.

Driver qla2x00tgt has 2 types of SCST sessions: local and not local. Sessions created by q2t_fc_port_added() are not local. Local sessions created if qla2x00tgt receives a command from remote initiator for which there is no know remote FC port and, hence, SCST session. Local sessions are created in tgt->sess_work (q2t_sess_work_fn()) by calling q2t_make_local_sess(). All received from remote initiators commands for local sessions are delayed until the sessions are created.

To minimize affecting initiators by FC fabric events, qla2x00tgt doesn't immediately delete SCST sessions scheduled for deletion, but instead delay them for some time. If during this time a command from an unknown remote initiator received, q2t_make_local_sess()/q2t_create_sess() at first check if a session for this initiator already exists and, if yes, undelete then reuse it after updating its s_id and loop_id to new values.

If a session not reused during the delete delay time, then q2t_del_sess_work_fn() asks the firmware internal database if it knows the corresponding remote FC port. If yes, then this session is undeleted and its s_id and loop_id updated to new values. If no, the session is deleted.

10.6 Handling stuck commands

Driver qla2x00tgt defines in tgt2x_template callback on_hw_pending_cmd_timeout for handling stuck commands in q2t_on_hw_pending_cmd_timeout() function, with max_hw_pending_time timeout set Q2T_MAX_HW_PENDING_TIME (60 seconds). If the firmware doesn't return reply for one or more IOCBs for the corresponding SCST command, SCST core calls this callback.

In this callback all the stuck commands are forcibly finished.

11. Debugging and troubleshooting

SCST core and its drivers provide excessive debugging and logging facilities suitable to catch and analyze problems of virtually any level of complexity.

Depending from amount debugging and logging facilities available, there are 3 types of builds:

Switch between build modes is done by calling "make x2y", where "x" - current build mode and "y" - desired build mode. For instance, to switch from release to debug mode you should run "make release2debug".

11.1 Logging levels management

Logging levels management is done using "trace_level" file located in the driver's proc interface subdirectory. Each SCST driver has it, except in the perf build mode. For instance, for SCST core it's located in /proc/scsi_tgt/. For qla2x00t it's located in /proc/scsi_tgt/qla2x00tgt/.

Reading from it you can find currently enabled logging levels.

You can change them by writing in this file, like:

# echo "add scsi" >/proc/scsi_tgt/trace_level

The following commands are available:

The following trace levels are common for all drivers:

The following trace levels are additionally available for SCST core:

11.2 Preparing a debug kernel

SCST logging can produce huge amount of logging, which default kernel configuration can't cope with, so it needs some extra adjustments.

For that you should change in lib/Kconfig.debug or init/Kconfig depending from your kernel version LOG_BUF_SHIFT from "12 21" to "12 25".

Then you should in your .config set CONFIG_LOG_BUF_SHIFT to 25.

Also, Linux kernel has a lot of helpful debug facilities, like lockdep, which allows to catch various deadlocks, or memory allocation debugging. It is recommended to enable them during SCST debugging.

The following options are recommended to be enabled (available depending from your kernel version): CONFIG_SLUB_DEBUG, CONFIG_PRINTK_TIME, CONFIG_MAGIC_SYSRQ, CONFIG_DEBUG_FS, CONFIG_DEBUG_KERNEL, CONFIG_DEBUG_SHIRQ, CONFIG_DETECT_SOFTLOCKUP, CONFIG_DETECT_HUNG_TASK, CONFIG_SLUB_DEBUG_ON, CONFIG_SLUB_STATS, CONFIG_DEBUG_PREEMPT, CONFIG_DEBUG_RT_MUTEXES, CONFIG_DEBUG_PI_LIST, CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_MUTEXES, CONFIG_DEBUG_LOCK_ALLOC, CONFIG_PROVE_LOCKING, CONFIG_LOCKDEP, CONFIG_LOCK_STAT, CONFIG_DEBUG_SPINLOCK_SLEEP, CONFIG_STACKTRACE, CONFIG_DEBUG_BUGVERBOSE, CONFIG_DEBUG_VM, CONFIG_DEBUG_VIRTUAL, CONFIG_DEBUG_WRITECOUNT, CONFIG_DEBUG_MEMORY_INIT, CONFIG_DEBUG_LIST, CONFIG_DEBUG_SG, CONFIG_DEBUG_NOTIFIERS, CONFIG_FRAME_POINTER, CONFIG_FAULT_INJECTION, CONFIG_FAILSLAB, CONFIG_FAIL_PAGE_ALLOC, CONFIG_FAIL_MAKE_REQUEST, CONFIG_FAIL_IO_TIMEOUT, CONFIG_FAULT_INJECTION_DEBUG_FS, CONFIG_FAULT_INJECTION_STACKTRACE_FILTER.

11.3 Preparing logging subsystem

It is recommended that you system logger daemon on the target configured:

11.4 Decoding OOPS messages

You can decode an OOPS message to the corresponding line in C file using gdb "l" command. For example, an OOPS message has a line:

[<ffffffff88646174>] :iscsi_scst:iscsi_extracheck_is_rd_thread+0x94/0xb0

You can decode it by:

$ gdb iscsi-scst.ko
(gdb) l *iscsi_scst:iscsi_extracheck_is_rd_thread+0x94

For that the corresponding module (iscsi-scst.ko) should be build with debug info. But modules not always have debug info built-in. To workaround it you can add "-g" flag in the corresponding Makefile (without changing anything else!) or enable in .config using "make menuconfig" building kernel with debug info. Then rebuild only the .o file you need.

For instance, to decode OOPS in mm/filemap.c in the kernel you need enable in .config building kernel with debug info and then run:

$ make mm/filemap.o
...
$ gdb mm/filemap.o