NAME
enbd-server − ENBD service daemon
SYNOPSIS
enbd-server ctrlport export−file [ export−files ] [ [ −s ] size−of−export ] [ −l01 ] [ −r ] [ −y ] [ −w order-timeout ] [ −i signature ] [ −b blocksize ] [ −t negotiation−timeout ] [ −F minport[-maxport] ] [ −K ] [ −n ] [ −a ] [ −h cache_limit ] [ −cert certificate_file ] [ −keyfile key_file ] [ −CApath certificate_authority_dir ] [ −CAfile certificate_authority_file ] [ −verify depth ]
DESCRIPTION
The enbd-server program is a Network Block Device (NBD) service daemon that handles a remote clients requests for a local resource. The local resource may be a file (or files) and/or block devices. If there is more than one local resource, they may be aggregated using RAID−style protocols. Currently RAID levels 0 (linear mode) and 1 (striping) are supported. Aggregated resources may together exceed the maximum file system size limit (currently 2GB on linux intel) so long as their components are each within the limit.
The enbd-server operates as a normal user-level process and can run on any unix operating system. It makes no special ioctl() calls. The implementation allows the clients read-only or read-write access to the resource(s). Communication is demand-multiplexed over several channels and different channels may pass via different physical routes, depending on how the client set up the connection.
At startup, the server will negotiate for a handshake with the client on the specified port within the given timeout interval, or forever if the timeout is zero. If the handshake is successful and the connection is made successfully, the channels within the connection will be maintained, checked, renegotiated and recovered automatically if they break. Recovery will be transparant to the user if at least one working connection remains available at all times. The daemon starts a process for each connection and that process watchdogs subprocesses for each channel, restarting them if they die.
The server and client will use Secure Socket Layer (SSL) encryption and authentification procedures if any of the SSL−associated options appear in the command line.
OPTIONS
|
−l |
This option specifies that RAID linear mode will be used to aggregate multiple local resources. Resources are aggregated in the order in which they appear on the command line. The limit is 16TB with 2GB components on ia32. | ||
|
−0 |
This option specifies that RAID striping mode will be used to aggregate multiple local resources. The limit is 16TB with 2GB components on ia32. | ||
|
−1 |
This option specifies that RAID mirroring mode will be used to aggregate multiple local resources. Mirroring is currently limited to 2GB on ia32. | ||
|
−r |
This option specifies that local resources are opened in read-only mode. | ||
|
−y |
This option specifies that local resources are opened in synchronous mode for writes. |
−i signature
This option specifies an ascii "signature" that will be shared between server and client. It functions as a session identifier in the absence of SSL authentification and encryption (see below). The signature must be at most 128 characters long. If not specified, a semi-random signature will be generated from the host IP and resource file names. Not specifying a signature is a GOTCHA that can lead to behavior unexpected by the unwary user. It forms part of the name of the pidfile generated by the server. Please specify a signature.
−s size
This option coverrides the size of the exported resource, for use in case they are not detectable by the internal probes (e.g. removable media). The size can be a comma separated list, in case of multiple resources. An invalid numerical string, such as "-", in any position will cause the size of that resource to be probed for. Unit multipliers "K", "M" and "G" are understood.
−b blocksize
This option specifies the size of the transaction unit, in bytes. Valid values are 1024, 2048, 4096. RAIDed local resources are striped in these units, or truncated to a multiple of this unit size. The server and client negotiate a common blocksize at startup that is the maximum of the two offers, so the blocksize chosen may not be the one specified here. That may be significant for restarted striped local resources. The default is 1024.
−F minport-maxport
When this argument is specified it indicates that the server will confine its search for a free session port to the given range. If this argument is not present it will start searching at the port following the control port (i.e. the one the server is listening on). A range x−y may be given, or just the start of the range, x, or just the end of the range, −y.
−t negotiation timeout
This option specifies the maximum duration of the initial negotiations, in seconds. After this timeout the server will retry negotiation until the timeout elapses again. The cycle continues until contact is established. If contact is irretrievably lost at some stage after contact has been correctly established, the server will reenter the negotiation phase, retrying every timeout seconds if it does not get all the way through the negotiation. The default timeout is 90 seconds.
−w write order timeout
This option specifies how long (in ms) the server should wait for a missing write in the sequence of write commands from the client. The default is 0ms (i.e. never wait). With a value of 1000 the server will delay writes to disk for up to 1s if a sequence number is missed. This option can be important if the overlying file system is a journalling file system, as it guarrantees coherency of the image on the server side at all times.
−cert certificate_file
This option declares the full path to an SSL certificate identifying the server machine. The certificate should have been issued by a Certificate Authority recognized by both server and client machines, even if the certificates are self-signed (see the openSSL documentation at www.openssl.org for details).
−keyfile key_file
This option declares the full path to a file containing the private key that matches the certificate_file specified in the −cert option. If this option is not given (but −cert is), the certificate_file itself will be searched for the private key. Both files are assumed to be in PEM format.
−CApath Certificate_Authority_dir
This option specifies the directory in which the identifying certificates (needed for verifying the certificate from the client machine) of recognized Certificate Authorities are kept.
−CAfile Certificate_Authority_certificate
This option declares the full path to the certificate of the Certificate Authority (needed for verifying the certificate from the client machine).
−verify n
This option turns on authentication of the client certificate if n is greater than 0. The value of n specifies how far to go during the SSL verification procedure. 1 is suficient for self-signed certificates.
EXPERIMENTAL OPTIONS
|
−a |
This option tells the server to run asynchronously with respect to the network. It will acknowledge writes from the client before it has written them to disk. There is a potential to lose data here. This is faster than the normal service mode and it can avoid a deadlock under some situations. For example two servers with two clients both writing to each other at the same time will deadlock when both kernels are simultaneously full with dirty buffers aimed at the clients. Each nbd server wants to write to disk to relieve the pressure but cannot get buffers to do so because they are locked up in its client. This happens localhost to loaclhost too. Using a raw device as resource or persuading the server to open the device in direct i/o mode (how?) should also achieve the same end. | ||
|
−K |
This option tells the server to lock regions of the resource as requests come in for them. The locking permits multiple readers of a zone or single writers, according as the request is read or write. The option should make no difference. Normally off. | ||
|
−n |
This option tells the server to open the resource in O_DIRECT mode, thus bypassing the host kernel’s virtual memory system. This might improve thruput, or at least make it less bursty. Normally off. Problematic on 2.4 kernels. |
−hcache_limit
This option controls the way the server caches its old replies, in case it gets asked a request twice. The default value is some large number (the real limit will be the number of results and associated data that fits in 8KB). Set to 0 to turn off result caching altogether.
SIGNALS
enbd-server
recognizes the following signals:
SIGTERM
causes enbd-server to shut down connections and prepare to die.
|
SIGHUP |
forces a reconnection attempt. |
SIGKILL
definitively shuts up the
server. A useful termination sequence is
killall enbd−server; killall -9
enbd−server
FILES
The server leaves its PID in /var/run/nbd-server-ID.pid where ID is the identifier given with −i. It will not start if this file contains the PID of a running process.
The server also maintains a list of known client IP addresses in /var/state/nbd/server-ID.client_ips. This list is persistent through reboots, and the server will contact the listed addresses at restart on their enbd-cstatd service (see services(5)), passing across the servers IP address and control port.
When contacted, the enbd-cstatd on the client will deliver SIGPWR to any still running daemons connected to our port, and will restart daemons that should be connected to our port according to its enbd.conf(5).
SEE ALSO
enbd-client(8), enbd.conf(5), www.openSSL.org
NOTES
The server need
not run as root if the port chosen is above 1024. An example
command line is
enbd-server 4017 /a/f1 /a/f2 -1 -b 4096 -i nbd
or with SSL
options:
enbd-server ... -CAfile /b/cacert.pem -cert
/b/mycert.pem
BUGS
It is not clear to me (ptb) if serving whole raw-partitions greater that 2GB in size works. It probably depends on the kernel version. It seems to work fine in kernels 2.2.18 and 2.4.0.
The server (as of 2.4.24) only permits one connection per client IP address. Subsequent connections kill off earlier ones. It should be per IP plus client identifier, but I’ve been too lazy to set up the pipe back from the session master to the generic server daemon that would tell it what the client id that it got was.
AUTHORS
Pavel Machek wrote the orignal nbd suite, with contributions from Stephen Tweedie. Peter Breuer ported it to linux 2.0. Peter Breuer added multiplexing and renegotiation. Andres Marin moved networking into user space and added SSL.