NAME
enbd-client − ENBD client daemon
SYNOPSIS
enbd-client
{ host ctrlport | host:ctrlport }
{ route ... | −n nchan }
[ −a ]
[ −b blocksize ]
[ −e ]
[ −i identifier ]
[ −m ]
[ −p pulse−intvl ]
[ −r ]
[ −c ]
[ −s ]
[ −t negotiation−timeout ]
[ −x max−negotiation−retries ]
[ −cert certificate_file ]
[ −key key_file ]
[ −CApath certificate_authority_dir ]
[ −CAfile certificate_authority_file ]
[ −verify depth ]
device
DESCRIPTION
The enbd-client program is a Network Block Device (NBD) client daemon that handles kernel requests made on a local NBD and passes them across the net to a remote server daemon (see enbd−server(8)). The result is that a remote resource on the server appears local. The remote resource is effectively proxied by the NBD. NBDs may be treated like any other block device. In particular, they may have file systems made on them and then may be mounted into the local file system tree.
|
The enbd-client operates as a normal user-level process but communicates with the linux kernel via ioctl() calls. The enbd.o module must be present in the kernel to support it. |
The client demand-multiplexes communications to the remote server over the routing aliases defined on the command line. More than one routing alias may be given, and at least one must be given. It is normal to use the IP address or name of the remote host as the routing alias. But if several are specified and they go through different physical interfaces, then available bandwidth is increased proportionately. |
It is a good idea to name even a single available route several times on the commandline, because then classic pipelining parallelism occurs inside the several daemons which will start up, which speeds up processing noticably and provides increased redundancy. See section NOTES for commandline examples. |
At startup, the client will negotiate a connection with the server within the specified timeout, or try forever if zero is specified. When initialization is successful, the connection will be maintained, checked, renegotiated and recovered automatically if it breaks. Recovery will be transparant to the user if at least one routing alias remains in order at all times. The daemon starts a subprocess for each routing alias named on the command line and watchdogs the subprocesses, restarting them if they die.
The client and server will use Secure Socket Layer (SSL) encryption and authentification procedures if any of the SSL−associated options appear in the command line.
ARGUMENTS
host
ctrlport route [ route .. ]
host:ctrlport route [ route .. ]
host:ctrlport -n nchan
The first argument is the name
of a server host to connect to for control information and
the second argument is the port number at which to connect.
Subsequent arguments define channels to the server per data
connection. Each argument must be a valid IP alias or DNS
alias of the server (and is usually a repeat of the
host parameter). At least one routing alias must be
given, normally host. Multiple aliases, and hence
multiple channels within the connection, provide redundancy
and parallelism, with increased throughput, so at least two
should be used per connection.
In the case that all routing aliases are repeats of
host, the shorthand "−n nchan" form
may be used. E.g. "-n 1" for just one channel
within the connection (this is the default).
To set up channels across different routes to the server,
use different IP aliases for the server as the routing
aliases and route them differently. That will increase
bandwidth proportionately.
|
device |
The pathname to the ENBD "whole device" (/dev/nda, /dev/ndb, etc.) The standard ENBD naming scheme (see section NOTES) must be implemented on the client. In the case of kernel devfs being in charge of the /dev directory structure, the device should be given as just /dev/nd/a, /dev/nd/b, etc., and the appropriate block device specials in these directories will be picked up correctly. |
OPTIONS
The following
options are always available (see later section for
EXPERIMENTAL OPTIONS).
−b blocksize
This option specifies the size of the transaction unit, in bytes. Valid values are 1024, 2048, 4096. The client and server negotiate a common blocksize at startup that is the maximum of the two offers. A larger value is appropriate to a good network connection. The default is 1024.
|
−e |
This option specifies that the ENBD device will transmit network errors almost immediately to any higher level device, such as RAID, that may want to be notified of them. The flag is normally absent and the ENBD device will hide errors internally, blocking new requests until they can be met. | ||
|
The default (i.e. flag absent) is the correct configuration for a stand-alone ENBD device, as it will normally have several redundant channels available and/or may be subject to intermittent network delays and failures that should not be expected to take the device offline. But if the device is overlain by another (RAID, LVM, ...) then the responsibility for treating errors will better be handed off to the upper device, and this flag does that. | |||
|
You may want to search out the FR1 patch for linux raid1 on the net, since it makes the linux multidisk support substrate talk to the enbd device, and get all this kind of thing right on its own. |
−i identifier
This option specifies a string that will be used by the client when identifying itself to the server, in order that different clients with the same IP address may be distinguished. It is also used in setting up the clients pidfile and by the clientside state-monitoring code, so you should set it.
|
If you do not specify it, a secret random short printable string will be generated, and it will be very hard to recreate the string in case you ever want to reconnect under the same identifier again, leading to what, to you, may seem strange behaviour! |
−p pulse_intvl
Specify the interval in seconds between heartbeat pulses to the server which determine if it is alive and well when no other activity is present. Default 10s. This number also determines internal timeouts for data and control communications in general, which will be three times the number specified here.
|
Note that the pulse also causes the server side to check the condition of its resource, so if the server resource is a removable media device such as a floppy or cdrom, you probably want to reduce the pulse interval to two or three seconds. | |||
|
−r |
This option specifies that the ENBD device will be readonly. It overrides the readonly state communicated by the server. Do not specify −r if you intend to cache writes client-side (see −c option). The correct combination for that is −c on the client daemon and −r on the server daemon. That will specify that the remote resource is readonly and the local device is readwrite, with writes being cached locally. | ||
|
−c |
This option makes the ENBD device cache writes locally, instead of passing them over to the remote server. Subsequent reads will see the locally written data, where appropriate. The kernel is responsible for storing the data, and the stored data uses memory. This is intended to be used for root devices which will be copied off a readonly remote server and modified slightly on the local node. | ||
|
−s |
This option locks the client in memory, allowing the use of the nbd device as swap. You’re crazy if you want to swap over the net, but if you’re crazy that way, try this flag. |
−t negotiation−timeout
This option specifies the duration in seconds of the period for which the client waits for connections to the server to become established. On error or failure the client will wait a few seconds, then retry until the timeout elapses. If 0 is specified here, the client will try forever. The default is 120 seconds.
−x max−negotiation−retries
This option specifies the number of times the client will revive itself after a failed negotiation. If 0 is specified here, the client will retry forever. The default is 0.
SSL OPTIONS
The following
options are available when the client daemon (and server)
are compiled for Secure Socket Layer transports. The use of
any one of these options on the command line then makes the
daemon use an SSL transport instead of plain TCP .
−cert certificate_file
This option declares the full path to an SSL certificate identifying the server machine. The certificate should have been issued by a Certificate Authority recognized by both server and client machines, even if the certificate is self-signed (see the openSSL documentation at www.openssl.org for full details of the requirements and issuing procedure).
−key key_file
This option declares the full path to a file containing the private key that matches the certificate specified by the −cert option. If this option is not specified (but −cert is), the certificate_file itself will be searched for the private key. Both files are assumed to be in PEM format.
−CApath Certificate_Authority_dir
This option specifies the directory in which identifying certificates (needed for verifying the certificate from the server machine) from recognized Certificate Authorities are kept.
−CAfile Certificate_Authority_certificate
This option declares the full path to the certificate of the Certificate Authority (needed for verifying the certificate from the server machine).
−verify n
This option turns on authentication of the server certificate if n is greater than 0. The precise value of n specifies how far to go in the verification procedure (see the openSSL documentation at www.openssl.org for details). 1 is sufficient for self-signed certificates.
EXPERIMENTAL OPTIONS
The following options are largely untested and "unofficial" but are documented here for completeness.
|
−a |
This option makes the client asynchronous (and unreliable). It will send an ack to the kernel before receiving an ack from the server. There are situations in which this behaviour may be desirable, for example, when running against localhost, or when running swap over the device. Do not use this option without thorough testing. At least it currently works! | ||
|
−m |
This option starts the client up in "trick" mode to help overlying RAID resyncs. It generates an md5sum for the data that it will write to the server, and requests the servers corresponding md5sum before sending. If the two md5sums coincide, the write is skipped. The mode is disabled automatically when 10 consecutive distinct md5summed requests occur. It is also reenabled automatically every 1000 requests. The numbers 10 and 1000 can be changed via the /proc/sys/dev/nbd/*threshold files. This option is safe. |
SIGNALS
enbd−client
recognizes the following signals:
SIGTERM
causes enbd−client
to shut down connections and prepare to die. A useful
termination sequence is
killall -USR1 enbd−client; killall -TERM enbd−client
|
SIGHUP |
forces a reconnection attempt. |
SIGUSR1
errors out pending requests on
the device and keeps it in the disabled state for 5s, after
which it enables the device again. During this period all
new requests on the device will be errored. This is the
appropriate state to put the device in in order to take it
out of a RAID array cleanly. If there are no daemons left
alive to receive USR1, do
echo -n 0 > /proc/nbdinfo
instead. Use USR1 if a sync(8) or umount(8) is stuck.
SIGUSR2
causes the client to forcefully
shutdown the device and attempt to return the kernels
internal state to sanity. Do not try this before trying USR1
first, because USR2 will reset the module usage counts to
zero, which may not match reality and hence may cause tears
later. For example, you will be able to remove the kernel
module after USR2, but you may not want to, since the device
may still be mounted and the kernel will later oops when you
try and unmount the (vanished) device. Use umount(8) and
fuser(1) -k before USR2. If there are no daemons left alive
to receive USR2, do
echo -n 1 > /proc/nbdinfo
instead.
SIGKILL
definitively shuts up the client. But it won’t get any chance to tell the kernel about its impending demise or clean up its PID file.
|
SIGPWR |
causes the client daemon to shutdown and start from scratch, as though TERM and then an exec had been issued. Use on the client after the server has been rebooted after power failure. It indicates that a new session is needed. |
NOTES
The client has
to run as root in order to access the ENBD. An example
command line that starts a client identified as X0
with two connections of blocksize 4096 to server1 on
port 4017 is
enbd-client server1:4017 /dev/nda -n 2 -i X0 -b 4096
or with SSL
options:
enbd-client ... -CAfile /b/cacert.pem -cert
/b/mycert.pem
The ENBD is a partitioned device, but access is via the whole device only. The major number is 43. The partitions (up to 15 per whole device) serve as roots for the subprocesses started by the main daemon, as well as partitions in the normal sense (if the device is partitioned by fdisk(8)) The following sh(1) code will make the devices (the naming scheme is important):
for i in a b
c d; do
i_nr=’expr index abcd $i − 1’
mknod /dev/nd$i b 43 $[ $i_nr \* 16 ]
for j in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do
mknod /dev/nd$i$j b 43 $[ $i_nr \* 16 + $j ]
done
done
The enbd.o module must be inserted in the kernel before launching the client or using the devices. The devices named nda, ndb, etc. are "whole devices" and support block i/o. The devices named nda1, nda2, etc. are "partitions" and are used exclusively by the client daemons for their control ioctls.
If devfs support is compiled into the kernel and activated on the /dev subdirectory, then the naming scheme changes somewhat. The clients will look for and detect and work with either scheme. With devfs, the "whole device" is named /dev/nd/a/0, /dev/nd/b/0, etc. The "partitions" are /dev/nd/a/1, /dev/nd/a/2, etc.
The "whole device" can be formatted and mounted like any other block device. E.g.
mke2fs /dev/nda; mount /dev/nda /mnt
or, in the presence of defvs:
mke2fs /dev/nd/a/0; mount /dev/nd/a/0 /mnt
FILES
The kernel
module creates a file /proc/nbdinfo showing the
status of the devices. The most useful data are the counts
of blocks transferred, pending, or errored, per port. The
file can also be written to.
The driver also accepts commands through this file. See the
driver source code (the write_proc function) for a
definitive list of commands accepted, but the list includes,
for example:
merge_requests=1
and
merge_requests=0
to turn on and off, respectively, request aggregation in the kernel for the nbd device. In general, any module parameter that can be supplied on the insmod(8) line for the module can also be changed dynamically by writing the same command to the /proc file. For best results suppress a trailing newline, as the parser in the driver is limited.
The client records its PID in /var/run/enbd-client-ID.pid where ID is the identifier given via −i.
The client is usually started by a /etc/init.d/enbd script which in turn references the entries in /etc/enbd.conf. The latter is also used by the enbd-cstatd daemon if it is installed. That daemon usually runs out of inetd(8) and serves to maintain connections across reboots. It is invoked by the server on the remote machine when the remote server reboots, for example, and it will look in /etc/enbd.conf and either start the appropriate enbd-client or deliver SIGPWR to a running client.
SEE ALSO
AUTHORS
Pavel Machek wrote the orignal nbd suite, with contributions from Stephen Tweedie. Peter Breuer ported it to linux 2.0. Peter Breuer added multiplexing and renegotiation. Andres Marin moved networking into user space and added SSL. Peter Breuer ported it to linux 2.2 and 2.4, and added caching and other features.