From a586e65bbd017ab55fe4149dd1bcba5c3a72bcd1 Mon Sep 17 00:00:00 2001 From: Michael S. Tsirkin Date: Sun, 15 Nov 2015 21:25:11 +0200 Subject: vhost-user: update spec description Clarify logging setup to make sure all clients comply in a way that is future-proof. Document how rings are started/stopped. Signed-off-by: Michael S. Tsirkin Reviewed-by: Victor Kaplansky --- docs/specs/vhost-user.txt | 64 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 55 insertions(+), 9 deletions(-) (limited to 'docs') diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt index 26dde2ec42..df40cec54e 100644 --- a/docs/specs/vhost-user.txt +++ b/docs/specs/vhost-user.txt @@ -87,6 +87,14 @@ Depending on the request type, payload can be: User address: a 64-bit user address mmap offset: 64-bit offset where region starts in the mapped memory +* Log description + --------------------------- + | log size | log offset | + --------------------------- + log size: size of area used for logging + log offset: offset from start of supplied file descriptor + where logging starts (i.e. where guest address 0 would be logged) + In QEMU the vhost-user message is implemented with the following struct: typedef struct VhostUserMsg { @@ -138,6 +146,23 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Starting and stopping rings +---------------------- +Each ring is initialized in a stopped state, client must not process it until +ring is enabled. + +If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, client must start and +stop ring processing upon receiving VHOST_USER_SET_VRING_ENABLE with parameters +1 and 0 respoectively. + +If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, client must start +ring processing upon receiving a kick (that is, detecting that file descriptor +is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK, and stop +ring processing upon receiving VHOST_USER_GET_VRING_BASE. + +While rings are running, client must support changing some configuration +aspects on the fly. + Multiple queue support ---------------------- @@ -162,9 +187,13 @@ the slave makes to the memory mapped regions. The client should mark the dirty pages in a log. Once it complies to this logging, it may declare the VHOST_F_LOG_ALL vhost feature. +To start/stop logging of data/used ring writes, server may send messages +VHOST_USER_SET_FEATURES with VHOST_F_LOG_ALL and VHOST_USER_SET_VRING_ADDR with +VHOST_VRING_F_LOG in ring's flags set to 1/0, respectively. + All the modifications to memory pointed by vring "descriptor" should be marked. Modifications to "used" vring should be marked if -VHOST_VRING_F_LOG is part of ring's features. +VHOST_VRING_F_LOG is part of ring's flags. Dirty pages are of size: #define VHOST_LOG_PAGE 0x1000 @@ -173,22 +202,35 @@ The log memory fd is provided in the ancillary data of VHOST_USER_SET_LOG_BASE message when the slave has VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol feature. -The size of the log may be computed by using all the known guest -addresses. The log covers from address 0 to the maximum of guest +The size of the log is supplied as part of VhostUserMsg +which should be large enough to cover all known guest +addresses. Log starts at the supplied offset in the +supplied file descriptor. +The log covers from address 0 to the maximum of guest regions. In pseudo-code, to mark page at "addr" as dirty: page = addr / VHOST_LOG_PAGE log[page / 8] |= 1 << page % 8 +Where addr is the guest physical address. + Use atomic operations, as the log may be concurrently manipulated. +Note that when logging modifications to the used ring (when VHOST_VRING_F_LOG +is set for this ring), log_guest_addr should be used to calculate the log +offset: the write to first byte of the used ring is logged at this offset from +log start. Also note that this value might be outside the legal guest physical +address range (i.e. does not have to be covered by the VhostUserMemory table), +but the bit offset of the last byte of the ring must fall within +the size supplied by VhostUserLog. + VHOST_USER_SET_LOG_FD is an optional message with an eventfd in ancillary data, it may be used to inform the master that the log has been modified. -Once the source has finished migration, VHOST_USER_RESET_OWNER message -will be sent by the source. No further update must be done before the -destination takes over with new regions & rings. +Once the source has finished migration, rings will be stopped by +the source. No further update must be done before rings are +restarted. Protocol features ----------------- @@ -259,11 +301,13 @@ Message types * VHOST_USER_RESET_OWNER Id: 4 - Equivalent ioctl: VHOST_RESET_OWNER Master payload: N/A - Issued when a new connection is about to be closed. The Master will no - longer own this connection (and will usually close it). + This is no longer used. Used to be sent to request stopping + all rings, but some clients interpreted it to also discard + connection state (this interpretation would lead to bugs). + It is recommended that clients either ignore this message, + or use it to stop all rings. * VHOST_USER_SET_MEM_TABLE @@ -388,6 +432,8 @@ Message types Master payload: vring state description Signal slave to enable or disable corresponding vring. + This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES + has been negotiated. * VHOST_USER_SEND_RARP -- cgit v1.2.3-55-g7522 From 7ebcfe569211f6ff5402b558b85e2ce1e1066cf6 Mon Sep 17 00:00:00 2001 From: Michael S. Tsirkin Date: Tue, 17 Nov 2015 13:55:48 +0200 Subject: specs/vhost-user: fix spec to match reality We wanted to start/stop rings on VRING_ENABLE, but that is not what QEMU does. Rather than tweaking code some more, with risk to stability, let's just document it as it is. We'll be able to fix this in the future with a new protocol feature bit. Reported-by: Victor Kaplansky Signed-off-by: Michael S. Tsirkin --- docs/specs/vhost-user.txt | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) (limited to 'docs') diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt index df40cec54e..7b9cd6d0dd 100644 --- a/docs/specs/vhost-user.txt +++ b/docs/specs/vhost-user.txt @@ -148,20 +148,26 @@ a feature bit was dedicated for this purpose: Starting and stopping rings ---------------------- -Each ring is initialized in a stopped state, client must not process it until -ring is enabled. +Client must only process each ring when it is both started and enabled. + +If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, the ring is initialized +in an enabled state. -If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, client must start and -stop ring processing upon receiving VHOST_USER_SET_VRING_ENABLE with parameters -1 and 0 respoectively. +If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, the ring is initialized +in a disabled state. Client must not process it until ring is enabled by +VHOST_USER_SET_VRING_ENABLE with parameter 1, or after it has been disabled by +VHOST_USER_SET_VRING_ENABLE with parameter 0. + +Each ring is initialized in a stopped state, client must not process it until +ring is started, or after it has been stopped. -If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, client must start -ring processing upon receiving a kick (that is, detecting that file descriptor -is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK, and stop -ring processing upon receiving VHOST_USER_GET_VRING_BASE. +Client must start ring upon receiving a kick (that is, detecting that file +descriptor is readable) on the descriptor specified by +VHOST_USER_SET_VRING_KICK, and stop ring upon receiving +VHOST_USER_GET_VRING_BASE. -While rings are running, client must support changing some configuration -aspects on the fly. +While processing the rings (when they are started and enabled), client must +support changing some configuration aspects on the fly. Multiple queue support ---------------------- -- cgit v1.2.3-55-g7522