Linux 2.6.x driver installation instructions and notes for Tahoma Technology's PCI DR11-W interface Model 10118. Tahoma Technology (formerly IKON Corporation) 107 2nd Avenue, Seattle, WA, USA 98109 206.728.6465 http://www.tahomatech.com tahoma@tahomatech.com 5 October, 2005 documentation changes to match diriver mods: fixed 32 bit arg size for ioctl, max board # increased, dev nodes created only for boards actually found, 10116 and 10118 both supported About Tahoma Technology: Tahoma Technology has purchased certain of the assets of Ikon Corporation from IKON Office Solutions (IOS). That's the legal version. What has happened is that the people who started and operated Ikon Corporation initially and under IOS have purchased the division back. We continue the same product line and support. References to "Ikon" here and elsewhere are left in place for historical and compatibility reasons. Tahoma/Ikon code and documentation has been previously, and remains, in the public domain. Driver Installation & Notes: This version of the driver was tested under Linux 2.6 It is NOT compatible with earlier versions of Linux. The initial release of this driver was the first that did DMA direct from user buffers, using the kiobuf capability in Linux 2.3/2.4. This version moves the user buffer mapping code into the driver itself - since kiobufs are not available in 2.6x. If the kernel has been configured for greater than 4G of RAM, the driver will be compiled to use a kernel copy buffer instead of mapping user buffers for DMA. Copy buffer mode may also be forced by an install script option. In x86_64 systems, the available iommu is used to map high memory addresses to 32 bit address space, so a copy buffer is not necessary. Hopefully our HIGHMEM config detection won't trip on x86_64 systems. NOTE: the driver and install script now support both 10116 and 10118 boards. This document is written assuming a 10118, but it also applies to the 10116. Board types may be mixed in a system. The install script also attempts to autodetect the appropriate include directory: /lib/modules/`uname -r/build/inlcude and /usr/src/linux, in that order. Other install script options are described below. Other files provided in this release include: idr_driver.c code source module for the Linux PCI DR11-W driver idr_var.h structure definition file for the PCI DR11-W driver idr_reg.h register definition file for the PCI DR11-W driver idr_io.h ioctl command definition file for the PCI DR11-W driver - to be #include'd' in the calling program idr_changes.h change dates and descriptions & module ID macros for the driver Makefile simple makefile for the new kbuild system idr_install-2.6 used to compile and load driver module link.master.c very simple interprocessor link program (master end of link) that calls the DR11-W driver link.slave.c simple interprocessor link program (slave end of link) link.master.manual.c version of link master that demonstrates use of MANUAL mode idr.loop.test.c loop-back test source for PCI DR11-W board 118notes-2.6 information on using the PCI DR11-W driver The DR11 driver is a module that can be loaded into a running kernel. Before loading, the driver should be compiled on the target machine, and the object module placed in an appropriate /lib/module directory. idr_install-2.6 can be used to compile, load, and enable boot loading of the driver,or may be used to manually perform each of these steps individually. The idr_install-2.6 script provided can be used to compile the standard or debug version of the driver and place it in the correct directory. To compile the object and place in /lib/modules/`uname -r`/misc: (as super user) ./idr_install-2.6 compile To compile the debug version of the driver: (as super user) ./idr_install-2.6 compile debug Note that the debug version of the driver generates numerous printk() calls, and should only be used for diagnostic purposes! The idr_install-2.6 script provided will load the module and create the device node(s) in the /dev directory. Node names will be idr0, idr1, ... To load the driver module into a running kernel: (as super user) idr_install-2.6 load Once loaded, the driver will be available until unloaded or the system is rebooted. To cause the driver to be loaded when the system is booted: (as super user) idr_install-2.6 autoload This will create /etc/idr (if not already present) and copy idr_install-2.6 to /etc/idr. It also adds a line to the end of /etc/rc.d/rc.local which will invoke "/etc/idr/idr_install-2.6 load" at boot time. Directory and script permissions are set to rwx access for root, and read only for everyone else. When the install script is invoked via rc.local at boot time, standard and error outputs are logged to /etc/idr/idr_install-2.6.bootlog. In the event of problems with autoloading the driver at boot time, inspect this log for possible diagnostic information. NOTE that any changes made to the local copy of idr_install-2.6 after this point will not be reflected in the copy in /etc/idr which is invoked at boot time. Use the idr_install-2.6 "autoload" or "all" option to update the copy in /etc/idr. To remove the driver module from the kernel, delete the driver object from the modules directory, remove the rc.local entry, remove /etc/idr (if empty after removing idr_install-2.6*), and delete the device node(s): (as super user) idr_install-2.6 remove To remove, compile, load, and enable autoloading of the driver with a single command : (as super user) idr_install-2.6 all or (as super user) idr_install-2.6 all debug The script will try to determine the appropriate compile include directory: This is somewhat of a moving target. Tested for, in order, are: /lib/modules/`uname -r`/build/include and /usr/src/linux. If no directory in the list is detected, the compiler is allowed to use its default include directory (probably /usr/include). The default directory is usually not suitable for compiling driver modules, and will cause warnings and errors. The install script will allow for multiple boards up to a maximum set by the IDR_BOARDS variable. The current default is 4. Increase as necessary. The script will only create device nodes for boards actually found (as determined by idr devices found in /proc/interrupts. The device node(s) will be created with owner=bin group=bin and permissions=666. Other script variables that control the default configuration of the driver and board, as well as the behavior of the script itself may be modified. See the Driver Default Parameters section, below, for information on driver "defines". See the script code for script variable information. NOTES: Residual Byte Counts: It is a unix convention that the read and write routines return a count of the number of bytes actually transferred. Unfortunately, the Pci interface chip on the 10118 does not have a readable DMA range counter. Therefore, the DR11 driver always indicates that all bytes requested in a read or write call were transferred. Unless an error was encountered, this will in fact always be the case. Errors will be indicated by the driver returning an error code after a read or write call. In general, bytes lost in an aborted read or write cannot be re-sent successfully anyway, so the "actual" byte count would serve little purpose. There are some applications that do not know in advance how many bytes will be received by a read call. In this case, a read is issued with a count larger than the anticipated maximum. The read is typically terminated by an ATTENTION signal from the attached device. AT this time, the application can read the DR11 range counter (not the dma counter) to determine how many bytes were received. IOCTL parameters: The PCI DR11-W driver's ioctl() commands are compatible with Tahoma's SunOS and Solaris drivers for Sbus boards, and with the Solaris and Solaris 7 drivers for Pci boards. See the idr_io.h file and the 118notes-2.6 file for ioctl command descriptions. Note that the idr_io.h file for Linux is slightly different from the idr_io.h files provided with the drivers for Solaris 1 and 2. The old and new ....io.h files will generate the same ioctl commands, but are not interchangeable. Use the new idr_io.h file when compiling on a Linux machine. DR11 Driver IOCTL Calls: See the file called 118notes-2.6 and the idr_io.h file for info on driver calls and defaults. DR11 Byte Swapping: All 10118 boards use a PLX chip (9060 or 9080) that provides hardware byte swapping. The driver - Linux or Solaris - will default to the swap mode appropriate for the host architecture. DR11 Sample Link Programs: The link.master and link.slave programs implement a simple example of a DR11 to DR11 interprocessor link using the 10118 and driver. These programs are compatible with the equivalent Solaris programs for the Sbus and Pci boards and drivers, and with Tahoma's in house test link software for Tahoma's other DR11 emulator products. They may be used for testing, or as examples of the link protocol, and how to communicate with the driver. Also included on this diskette is a version of the link master program - link.master.manual - that runs in manual mode. This is a simple demonstration of how to use multiple read and write calls per DR11 block. This technique is useful in continuous data collection applications. A link connection is accomplished by cross-connecting J1 and J2 between two DR11 emulators (or the real thing!). J1 connects to J2, and J2 to J1. This can be done using short transition cables from the DR11s to a pair of ground-plane ribbon cables, or in the case of Tahoma's Pci or Sbus boards, by using a single round, jacketed, shielded link cable. DR11 Loop-back Test Program The idr.loop.test program can be used to perform a repeating loop-back test of the 10118. It requires that an appropriate loop-back cable be installed. This cable accomplishes a pin-for-pin connection between the two sides of the 80-pin high-density connector, or between J1 and J2 of the transition cable assembly. The test program first tests the three software timeout functions of the driver, and then loops indefinitely on several tests of the features of the board and driver. At the end of each 100 passes, it outputs a pass count. In the event of an error, it prints a brief message and exits. This test works correctly only on boards that can "un-swap" DMA bytes that are swapped by the pci-memory path. This includes all 10118s and later 10116 boards that use the PLX 9060SD or 9080 chips. Driver Default Parameters: Several driver defaults may be changed by editing the idr_install-2.6 script and reloading the driver. Many defaults may also be overridden by the application, using ioctl() calls. IDR_BOARDS: The driver init_module code will probe for the number of boards specified by IDR_BOARDS. The pci bios probe function will be called repeatedly until no further boards are found, or IDR_BOARDS is exhausted. Specifying a board count larger than the number of boards present will be somewhat wasteful of kernel memory, as each (potential) board requires a kmalloc'd state structure. The install script wil create as many device nodes as specified by IDR_BOARDS, regardless of the number of boards actually found. Node names will be idr0, idr1, ... The default value is 4. IDR_SPEED: The timing of the handshake used between the DR11 emulator and the attached device is set by the IDR_SPEED variable. Permissible values are 0, 1, 2, and 3, with 0 giving the fastest timing. In general, the fastest timing that gives reliable data transfers should be used. See the manual for details. IDR_DMA_TIME: IDR_RDY_TIME: IDR_ATTN_TIME: There are three software timeout clocks used by the driver during waits: a dma timer, a wait-for-ready timer, and a wait for ATTENTION timer. The defualts for these timers are set by the IDR_DMA_TIME, IDR_RDY_TIME, and IDR_ATTN_TIME variables, respectively. All default times are in seconds. IDR_BYTE_SWAP: The default byte ordering mode used during DMA transfers is set by the IDR_BYTE_SWAP variable. When 0, it gives un-swapped transfers. When one, bytes within 16 bit words will be swapped. IDR_BUSY_POL: IDR_CYCLE_POL: The default polarities of the BUSY and CYCLE REQUEST signals are determined by the IDR_BUSY_POL and IDR_CYCLE_POL variables. IDR_CYCLE_POL = 0 causes the CYCLE REQUEST input to assert on a rising edge; 1 selects falling edge active. IDR_BUSY_POL = 0 causes busy to be asserted low; 1 selects high assertion. These polarities should be selected to match the requirements of the attached device. IDR_WRITE_CYCLE: IDR_READ_CYCLE: At the beginning of each DR11 block transfer, some applications may require that the software cycle pulse be used to start the transfer (in link mode, it is required only for the sending end of the link). There are separate soft cycle variables for read and write operations. IDR_WRITE_CYCLE controls the write soft cycle, and IDR_READ_CYCLE controls reads. Setting either to 1 will cause a soft cycle pulse to be issued at the start of EACH corresponding type of DR11 block. IDR_READ_ACF2: Link mode also requires that the driver issue an ACLO FCN2 pulse (which becomes ATTENTION at the other end of the link) at the start of each DR11 read block. This tells the other end of the link that reads are enabled, and a cycle request from the sender would be honored. This pulse is enabled by setting the IDR_READ_ACF2 variable to 1. The default will be 0 for now - to stay consistent with the Sbus version of the driver. IDR_READ_F3: IDR_READ_F2: IDR_READ_F1: IDR_WRITE_F3: IDR_WRITE_F3: IDR_WRITE_F1: IDR_OPEN_F3: IDR_OPEN_F2: IDR_OPEN_F1: Several properties set the default states of the DR11 function bit outputs, FCN3, FCN2, and FCN1. This is done separately for read and write blocks, and when the device is opened. Each function bit has a separate property for each situation. The names are (hopefully) self- explanatory. Permitted valued are 0 or 1. IDR_MAX_PHYS_ORDER: This version of the driver does DMA directly to/from mapped and pinned user buffer when possible. It will do a maximum of 2**IDR_MAX_PHYS_ORDER * PAGE_SIZE bytes per DMA transfer. User buffers larger than this value will be transferred in multiple DMA blocks, transparently to the calling program. In the case of a HIGHMEM_64G (but not x86_64) enabled kernel a copy buffer of 2**IDR_MAX_PHYS_ORDER * PAGE_SIZE bytes will be allocated. In auto mode, one-time operations like pulsing GO, and setting the read or write function bits will be done at the beginning of the write or read, not with each DMA block, so the attached device will see what appears to be a single large block transfer of the size specified in the read or write call. In manual mode, GO, and function bits are not manipulated at all in a read or write operation, but left for the user to control via ioctl calls. Larger DMA transfers will improve overall efficiency somewhat, by minimizing the number of interrupts taken per user buffer (one per DMA block transfer). The tradeoff, of course, is the kernel memory and mapping resources used to map and pin the buffer, and build the scatter-gather list for the board's DMA mechanism. (On non-PC architectures - not tested yet - there will also be increased usage of iommu mapping resources). Given the quantities of cheap memory available on PC's, one should probably just choose a buffer size that keeps the attached device happy. IDR_MAX_PHYS_ORDER is specified in "orders". That is, 2**IDR_MAX_PHYS_ORDER * PAGE_SIZE gives the max transfer size in bytes. The default is 4, which with the current 4K page size gives 64k max bytes per transfer. Compile Options: IDR_DEBUG: Causes the driver to emit _many_ diagnostic printfs during operation. Do not use during normal production plotting! IDR_USER_DMA: IDR_USER_DMA = AUTO allows the driver to compile in user buffer DMA mode, as long as CONFIG_HIGHMEM_64G is not true. If the kernel supports > 4G of RMA (CONFIG_HIGHMEM_64G true) a copy buffer will be used - except in x86_64 systems. If IDR_USER_DMA = NO, a copy buffer will always be used. Load Options: IDR_FORCE_LOAD: If this option = YES, the -f (force) option will be used when the driver is loaded by insmod. This is probably not a good idea as it will ignore some important driver/kernel compatibility checks. It may be helpful during driver development.