summaryrefslogtreecommitdiffstats
path: root/qapi/block-core.json
Commit message (Collapse)AuthorAgeFilesLines
* qemu-img: Make unallocated part of backing chain obvious in mapEric Blake2021-07-121-2/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even when that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]; and the libnbd project was recently patched to give the human-readable name "absent" to an allocation-depth of 0. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset": listing). The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD. An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy. Signed-off-by: Eric Blake <eblake@redhat.com> Message-Id: <20210701190655.2131223-3-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [eblake: fix more iotest fallout] Signed-off-by: Eric Blake <eblake@redhat.com>
* block: Make blockdev-reopen stable APIAlberto Garcia2021-07-091-3/+3
| | | | | | | | | | This patch drops the 'x-' prefix from x-blockdev-reopen. Signed-off-by: Alberto Garcia <berto@igalia.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20210708114709.206487-7-kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* block: Support multiple reopening with x-blockdev-reopenAlberto Garcia2021-07-091-8/+10
| | | | | | | | | | [ kwolf: Fixed AioContext locking ] Signed-off-by: Alberto Garcia <berto@igalia.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20210708114709.206487-5-kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* block/rbd: Add support for rbd image encryptionOr Ozeri2021-07-091-2/+108
| | | | | | | | | | | | | | | | | | Starting from ceph Pacific, RBD has built-in support for image-level encryption. Currently supported formats are LUKS version 1 and 2. There are 2 new relevant librbd APIs for controlling encryption, both expect an open image context: rbd_encryption_format: formats an image (i.e. writes the LUKS header) rbd_encryption_load: loads encryptor/decryptor to the image IO stack This commit extends the qemu rbd driver API to support the above. Signed-off-by: Or Ozeri <oro@il.ibm.com> Message-Id: <20210627114635.39326-1-oro@il.ibm.com> Reviewed-by: Ilya Dryomov <idryomov@gmail.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* block/ssh: add support for sha256 host key fingerprintsDaniel P. Berrangé2021-06-301-1/+2
| | | | | | | | | | | | | | Currently the SSH block driver supports MD5 and SHA1 for host key fingerprints. This is a cryptographically sensitive operation and so these hash algorithms are inadequate by modern standards. This adds support for SHA256 which has been supported in libssh since the 0.8.1 release. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20210622115156.138458-1-berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Acked-by: Richard W.M. Jones <rjones@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* block: feature detection for host block supportJoelle van Dyne2021-06-251-4/+10
| | | | | | | | | On Darwin (iOS), there are no system level APIs for directly accessing host block devices. We detect this at configure time. Signed-off-by: Joelle van Dyne <j@getutm.app> Message-Id: <20210315180341.31638-2-j@getutm.app> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* block: Drop the sheepdog block driverMarkus Armbruster2021-05-121-92/+1Star
| | | | | | | | | It was deprecated in commit e1c4269763, v5.2.0. See that commit message for rationale. Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20210501075747.3293186-1-armbru@redhat.com> ACKed-by: Peter Krempa <pkrempa@redhat.com>
* block: Remove monitor command block_passwdMarkus Armbruster2021-03-231-14/+0Star
| | | | | | | | | | | | | | | | | Command block_passwd always fails since Commit c01c214b69 "block: remove all encryption handling APIs" (v2.10.0) turned block_passwd into a stub that always fails, and hardcoded encryption_key_missing to false in query-named-block-nodes and query-block. Commit ad1324e044 "block: remove 'encryption_key_missing' flag from QAPI" just landed. Complete the cleanup job: remove block_passwd. Cc: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20210323101951.3686029-1-armbru@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
* qapi/qom: Add ObjectOptions for throttle-groupKevin Wolf2021-03-191-0/+27
| | | | | | | | | | | | | | | | | | This adds a QAPI schema for the properties of the throttle-group object. The only purpose of the x-* properties is to make the nested options in 'limits' available for a command line parser that doesn't support structs. Any parser that will use the QAPI schema will supports structs, though, so they will not be needed in the schema in the future. To keep the conversion straightforward, add them to the schema anyway. We can then remove the options and adjust documentation, test cases etc. in a separate patch. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Acked-by: Paolo Bonzini <pbonzini@redhat.com> Acked-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
* block: remove 'dirty-bitmaps' field from 'BlockInfo' structDaniel P. Berrangé2021-03-181-10/+1Star
| | | | | | | The same data is available in the 'BlockDeviceInfo' struct. Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
* block: remove dirty bitmaps 'status' fieldDaniel P. Berrangé2021-03-181-45/+0Star
| | | | | | | The same information is available via the 'recording' and 'busy' fields. Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
* block: remove 'encryption_key_missing' flag from QAPIDaniel P. Berrangé2021-03-181-8/+0Star
| | | | | | | | | | This has been hardcoded to "false" since 2.10.0, since secrets required to unlock block devices are now always provided up front instead of using interactive prompts. Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
* qapi: backup: disable copy_range by defaultVladimir Sementsov-Ogievskiy2021-01-261-1/+1
| | | | | | | | | | | | | | | | Further commit will add a benchmark (scripts/simplebench/bench-backup.py), which will show that backup works better with async parallel requests (previous commit) and disabled copy_range. So, let's disable copy_range by default. Note: the option was added several commits ago with default to true, to follow old behavior (the feature was enabled unconditionally), and only now we are going to change the default behavior. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-19-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* qapi: backup: add max-chunk and max-workers to x-perf structVladimir Sementsov-Ogievskiy2021-01-261-1/+12
| | | | | | | | | | | | | Add new parameters to configure future backup features. The patch doesn't introduce aio backup requests (so we actually have only one worker) neither requests larger than one cluster. Still, formally we satisfy these maximums anyway, so add the parameters now, to facilitate further patch which will really change backup job behavior. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-11-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* qapi: backup: add perf.use-copy-range parameterVladimir Sementsov-Ogievskiy2021-01-261-1/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Experiments show, that copy_range is not always making things faster. So, to make experimentation simpler, let's add a parameter. Some more perf parameters will be added soon, so here is a new struct. For now, add new backup qmp parameter with x- prefix for the following reasons: - We are going to add more performance parameters, some will be related to the whole block-copy process, some only to background copying in backup (ignored for copy-before-write operations). - On the other hand, we are going to use block-copy interface in other block jobs, which will need performance options as well.. And it should be the same structure or at least somehow related. So, there are too much unclean things about how the interface and now we need the new options mostly for testing. Let's keep them experimental for a while. In do_backup_common() new x-perf parameter handled in a way to make further options addition simpler. We add use-copy-range with default=true, and we'll change the default in further patch, after moving backup to use block-copy. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-2-vsementsov@virtuozzo.com> [mreitz: s/5\.2/6.0/] Signed-off-by: Max Reitz <mreitz@redhat.com>
* qapi: block-stream: add "bottom" argumentVladimir Sementsov-Ogievskiy2021-01-261-4/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | The code already don't freeze base node and we try to make it prepared for the situation when base node is changed during the operation. In other words, block-stream doesn't own base node. Let's introduce a new interface which should replace the current one, which will in better relations with the code. Specifying bottom node instead of base, and requiring it to be non-filter gives us the following benefits: - drop difference between above_base and base_overlay, which will be renamed to just bottom, when old interface dropped - clean way to work with parallel streams/commits on the same backing chain, which otherwise become a problem when we introduce a filter for stream job - cleaner interface. Nobody will surprised the fact that base node may disappear during block-stream, when there is no word about "base" in the interface. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20201216061703.70908-11-vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* qapi: copy-on-read filter: add 'bottom' optionAndrey Shinkevich2021-01-261-1/+19
| | | | | | | | | | | | | | | Add an option to limit copy-on-read operations to specified sub-chain of backing-chain, to make copy-on-read filter useful for block-stream job. Suggested-by: Max Reitz <mreitz@redhat.com> Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [vsementsov: change subject, modified to freeze the chain, do some fixes] Message-Id: <20201216061703.70908-6-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* qapi: add filter-node-name to block-streamAndrey Shinkevich2021-01-261-0/+6
| | | | | | | | | | | | | Provide the possibility to pass the 'filter-node-name' parameter to the block-stream job as it is done for the commit block job. Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [vsementsov: comment indentation, s/Since: 5.2/Since: 6.0/] Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20201216061703.70908-5-vsementsov@virtuozzo.com> [mreitz: s/commit/stream/] Signed-off-by: Max Reitz <mreitz@redhat.com>
* block: introduce preallocate filterVladimir Sementsov-Ogievskiy2020-12-181-1/+19
| | | | | | | | | | | | | It's intended to be inserted between format and protocol nodes to preallocate additional space (expanding protocol file) on writes crossing EOF. It improves performance for file-systems with slow allocation. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20201021145859.11201-9-vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> [mreitz: Two comment fixes, and bumped the version from 5.2 to 6.0] Signed-off-by: Max Reitz <mreitz@redhat.com>
* qapi: Normalize version references x.y.0 to just x.yMarkus Armbruster2020-12-101-14/+14
| | | | | | | | | | We use x.y most of the time, and x.y.0 sometimes. Normalize for consistency. Reported-by: Eduardo Habkost <ehabkost@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20201118064158.3359056-1-armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
* block: Remove unused BlockDeviceMapEntryMarkus Armbruster2020-11-091-29/+0Star
| | | | | | | | | | | | | | | | | | | BlockDeviceMapEntry has never been used. It was added in commit facd6e2 "so that it is published through the introspection mechanism." What exactly introspecting types that aren't used for anything could accomplish isn't clear. What "introspection mechanism" to use is also nebulous. To the best of my knowledge, there has never been one that covered this type. Certainly not query-qmp-schema, which includes only types that are actually used in QMP. Not being able to introspect BlockDeviceMapEntry hasn't bothered anyone enough to complain in almost four years. Get rid of it. Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Eric Blake <eblake@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20201104165513.72720-3-mreitz@redhat.com>
* qapi/block-core: Improve MapEntry documentationMax Reitz2020-11-091-5/+13
| | | | | | | | | | | | | | | MapEntry and BlockDeviceMapEntry are kind of the same thing, and the latter is not used, so we want to remove it. However, the documentation it provides for some fields is better than that of MapEntry, so steal some of it for the latter. (And adjust them a bit in the process, because I feel like we can make them even clearer.) Signed-off-by: Max Reitz <mreitz@redhat.com> Message-Id: <20201104165513.72720-2-mreitz@redhat.com> Acked-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
* nbd: Add 'qemu-nbd -A' to expose allocation depthEric Blake2020-10-301-2/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | Allow the server to expose an additional metacontext to be requested by savvy clients. qemu-nbd adds a new option -A to expose the qemu:allocation-depth metacontext through NBD_CMD_BLOCK_STATUS; this can also be set via QMP when using block-export-add. qemu as client is hacked into viewing the key aspects of this new context by abusing the already-experimental x-dirty-bitmap option to collapse all depths greater than 2, which results in a tri-state value visible in the output of 'qemu-img map --output=json' (yes, that means x-dirty-bitmap is now a bit of a misnomer, but I didn't feel like renaming it as it would introduce a needless break of back-compat, even though we make no compat guarantees with x- members): unallocated (depth 0) => "zero":false, "data":true local (depth 1) => "zero":false, "data":false backing (depth 2+) => "zero":true, "data":true libnbd as client is probably a nicer way to get at the information without having to decipher such hacks in qemu as client. ;) Signed-off-by: Eric Blake <eblake@redhat.com> Message-Id: <20201027050556.269064-11-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
* block/nvme: Add driver statistics for access alignment and hw errorsPhilippe Mathieu-Daudé2020-10-231-1/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Keep statistics of some hardware errors, and number of aligned/unaligned I/O accesses. QMP example booting a full RHEL 8.3 aarch64 guest: { "execute": "query-blockstats" } { "return": [ { "device": "", "node-name": "drive0", "stats": { "flush_total_time_ns": 6026948, "wr_highest_offset": 3383991230464, "wr_total_time_ns": 807450995, "failed_wr_operations": 0, "failed_rd_operations": 0, "wr_merged": 3, "wr_bytes": 50133504, "failed_unmap_operations": 0, "failed_flush_operations": 0, "account_invalid": false, "rd_total_time_ns": 1846979900, "flush_operations": 130, "wr_operations": 659, "rd_merged": 1192, "rd_bytes": 218244096, "account_failed": false, "idle_time_ns": 2678641497, "rd_operations": 7406, }, "driver-specific": { "driver": "nvme", "completion-errors": 0, "unaligned-accesses": 2959, "aligned-accesses": 4477 }, "qdev": "/machine/peripheral-anon/device[0]/virtio-backend" } ] } Suggested-by: Stefan Hajnoczi <stefanha@gmail.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Acked-by: Markus Armbruster <armbru@redhat.com> Message-id: 20201001162939.1567915-1-philmd@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
* block: Convert 'block_resize' to coroutineKevin Wolf2020-10-091-1/+2
| | | | | | | | | | | | | | | | | | | block_resize performs some I/O that could potentially take quite some time, so use it as an example for the new 'coroutine': true annotation in the QAPI schema. bdrv_truncate() requires that we're already in the right AioContext for the BlockDriverState if called in coroutine context. So instead of just taking the AioContext lock, move the QMP handler coroutine to the context. Call blk_unref() only after switching back because blk_unref() may only be called in the main thread. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Message-Id: <20201005155855.256490-15-kwolf@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
* replay: introduce info hmp/qmp commandPavel Dovgalyuk2020-10-061-1/+2
| | | | | | | | | | | | | | | This patch introduces 'info replay' monitor command and corresponding qmp request. These commands request the current record/replay mode, replay log file name, and the instruction count (number of recorded/replayed instructions). The instruction count can be used with the replay_seek/replay_break commands added in the next two patches. Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru> Acked-by: Dr. David Alan Gilbert <dgilbert@redhat.com> Acked-by: Markus Armbruster <armbru@redhat.com> Message-Id: <160174520026.12451.13112161947433306561.stgit@pasha-ThinkPad-X280> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* migration: introduce icount field for snapshotsPavel Dovgalyuk2020-10-061-2/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | Saving icount as a parameters of the snapshot allows navigation between them in the execution replay scenario. This information can be used for finding a specific snapshot for proceeding the recorded execution to the specific moment of the time. E.g., 'reverse step' action (introduced in one of the following patches) needs to load the nearest snapshot which is prior to the current moment of time. This patch also updates snapshot test which verifies qemu monitor output. Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru> Acked-by: Markus Armbruster <armbru@redhat.com> Acked-by: Kevin Wolf <kwolf@redhat.com> -- v4 changes: - squashed format update with test output update v7 changes: - introduced the spaces between the fields in snapshot info output - updated the test to match new field widths Message-Id: <160174518865.12451.14327573383978752463.stgit@pasha-ThinkPad-X280> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* qapi: Create block-export moduleKevin Wolf2020-10-021-166/+0Star
| | | | | | | | | | | | Move all block export related types and commands from block-core to the new QAPI module block-export. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Message-Id: <20200924152717.287415-3-kwolf@redhat.com> Acked-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* qapi: Use rST markup for literal blocksPeter Maydell2020-09-291-7/+9
| | | | | | | | | | | | | | | | | | | | | | | There are exactly two places in our json doc comments where we use the markup accepted by the texi doc generator where a '|' in the first line of a doc comment means the line should be emitted as a literal block (fixed-width font, whitespace preserved). Since we use this syntax so rarely, instead of making the rST generator support it, instead just convert the two uses to rST-format literal blocks, which are indented and introduced with '::'. (The rST generator doesn't complain about the old style syntax, it just emits it with the '|' and with the whitespace not preserved, which looks odd, but means we can safely leave this change until after we've stopped generating texinfo.) Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Message-Id: <20200925162316.21205-11-peter.maydell@linaro.org> Signed-off-by: Markus Armbruster <armbru@redhat.com>
* qapi: Fix doc comment indentation againPeter Maydell2020-09-291-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | In commit 26ec4e53f2 and similar commits we fixed the indentation for doc comments in our qapi json files to follow a new stricter standard for indentation, which permits only: @arg: description line 1 description line 2 or: @arg: line 1 line 2 but because the script updates that enforce this are not yet in the tree we have had a steady trickle of subsequent changes which didn't follow the new rules. Fix the latest round of mis-indented doc comments. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Message-Id: <20200925162316.21205-2-peter.maydell@linaro.org> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Updated for commit 4c437254b807 and a83e24ba1a5] Signed-off-by: Markus Armbruster <armbru@redhat.com>
* qapi/: fix some comment spelling errorszhaolichang2020-09-171-2/+2
| | | | | | | | | | | I found that there are many spelling errors in the comments of qemu, so I used the spellcheck tool to check the spelling errors and finally found some spelling errors in the qapi folder. Signed-off-by: zhaolichang <zhaolichang@huawei.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20200917075029.313-10-zhaolichang@huawei.com> Signed-off-by: Laurent Vivier <laurent@vivier.eu>
* Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into stagingPeter Maydell2020-09-111-24/+34
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Block layer patches: - qemu-img create: Fail gracefully when backing file is an empty string - Fixes related to filter block nodes ("Deal with filters" series) - block/nvme: Various cleanups required to use multiple queues - block/nvme: Use NvmeBar structure from "block/nvme.h" - file-win32: Fix "locking" option - iotests: Allow running from different directory # gpg: Signature made Thu 10 Sep 2020 10:11:19 BST # gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6 # gpg: issuer "kwolf@redhat.com" # gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full] # Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6 * remotes/kevin/tags/for-upstream: (65 commits) block/qcow2-cluster: Add missing "fallthrough" annotation block/nvme: Pair doorbell registers block/nvme: Use generic NvmeBar structure block/nvme: Group controller registers in NVMeRegs structure file-win32: Fix "locking" option iotests: Allow running from different directory iotests: Test committing to overridden backing iotests: Add test for commit in sub directory iotests: Add filter mirror test cases iotests: Add filter commit test cases iotests: Let complete_and_wait() work with commit iotests: Test that qcow2's data-file is flushed block: Leave BDS.backing_{file,format} constant block: Inline bdrv_co_block_status_from_*() blockdev: Fix active commit choice block: Drop backing_bs() qemu-img: Use child access functions nbd: Use CAF when looking for dirty bitmap commit: Deal with filters backup: Deal with filters ... Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
| * blockdev: Fix active commit choiceMax Reitz2020-09-071-19/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it. This means that block-commit's @backing-file parameter is no longer allowed for such nodes, and that users will have to issue a block-job-complete command. Neither should pose a problem in practice, because this case was basically just broken until now. (Since this commit already touches block-commit's documentation, it also moves up the chunk explaining general block-commit behavior that for some reason was situated under @backing-file.) Signed-off-by: Max Reitz <mreitz@redhat.com>
| * mirror: Deal with filtersMax Reitz2020-09-071-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top). base_overlay is introduced so we can query bdrv_is_allocated_above() on it - we cannot do that with base itself, because a filter's block_status is the same as its child node, so if there are filters on base, bdrv_is_allocated_above() on base would return information including base. Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "target_backing_bs", because that is what it really refers to. Signed-off-by: Max Reitz <mreitz@redhat.com>
| * stream: Deal with filtersMax Reitz2020-09-071-3/+10
| | | | | | | | | | | | | | | | | | | | Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node). Signed-off-by: Max Reitz <mreitz@redhat.com>
* | qapi/block-core.json: Fix nbd-server-start docsPeter Maydell2020-09-071-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit eed8b6917832 added some new text to the nbd-server-start documentation in the wrong place. Since this is after the 'Returns:' line it's parsed as if it were part of the documentation of the "Returns:' information. Move it up to join the rest of the "documentation of the type as a whole" doc text. This doesn't look odd in the current HTML rendering, but the new QAPI-to-rST handling will complain about the indent level of the lines not matching up with the 'Returns:' line. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Message-Id: <20200810195019.25427-4-peter.maydell@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Markus Armbruster <armbru@redhat.com>
* | qapi: Fix indentation, againPeter Maydell2020-09-071-4/+4
|/ | | | | | | | | | | | | | | | | | | | | | | | In commit 26ec4e53f2 and similar commits we fixed the indentation for doc comments in our qapi json files to follow a new stricter standard for indentation, which permits only: @arg: description line 1 description line 2 or: @arg: line 1 line 2 Unfortunately since we didn't manage to get the script changes that enforced the new style in, a variety of commits (eg df4097aeaf71, 2e4457032105) introduced new doc text which doesn't follow the new stricter rules for indentation on multi-line doc comments. Bring those into line with the new rules. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Message-Id: <20200810195019.25427-3-peter.maydell@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Markus Armbruster <armbru@redhat.com>
* qapi/block-core.json: Remove stale description of 'blockdev-add'Kashyap Chamarthy2020-09-031-3/+1Star
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | On a 'qemu-discuss' thread[1], Kevin identifies that the current doc blurb for @blockdev-add is stale: This is actually a documentation bug. @id doesn't exist, blockdev-add never creates a BlockBackend. This was different in the very first versions of the patches to add blockdev-add and we probably just forgot to update the documentation after removing it. So remove the stale bits. And the requirement for 'node-name' is already mentioned in the documentation of @BlockdevOptions: [...] # @node-name: the node name of the new node (Since 2.0). # This option is required on the top level of blockdev-add. # Valid node names start with an alphabetic character and may # contain only alphanumeric characters, '-', '.' and '_'. Their # maximum length is 31 characters. [...] [1] https://lists.nongnu.org/archive/html/qemu-discuss/2020-07/msg00071.html -- equivalent to "-drive if=ide,id=disk0....." Fixes: be4b67bc7d ("blockdev: Allow creation of BDS trees without BB") Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com> Suggested-by: Kevin Wolf <kwolf@redhat.com> Message-Id: <20200805100158.1239390-1-kchamart@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
* qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bitAlberto Garcia2020-08-251-0/+7
| | | | | | | | | | | | | Now that the implementation of subclusters is complete we can finally add the necessary options to create and read images with this feature, which we call "extended L2 entries". Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <6476caaa73216bd05b7bb2d504a20415e1665176.1594396418.git.berto@igalia.com> [mreitz: %s/5\.1/5.2/; fixed 302's and 303's reference output] Signed-off-by: Max Reitz <mreitz@redhat.com>
* schemas: Add vim modelineAndrea Bolognani2020-08-031-0/+1
| | | | | | | | | | | | | | | | | | | | | | The various schemas included in QEMU use a JSON-based format which is, however, strictly speaking not valid JSON. As a consequence, when vim tries to apply syntax highlight rules for JSON (as guessed from the file name), the result is an unreadable mess which mostly consist of red markers pointing out supposed errors in, well, pretty much everything. Using Python syntax highlighting produces much better results, and in fact these files already start with specially-formatted comments that instruct Emacs to process them as if they were Python files. This commit adds the equivalent special comments for vim. Signed-off-by: Andrea Bolognani <abologna@redhat.com> Message-Id: <20200729185024.121766-1-abologna@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
* qapi: Delete unwanted indentation of top-level expressionsMarkus Armbruster2020-08-031-13/+13
| | | | | | | Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20200730091656.2633334-1-armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> [One more line de-indented]
* Remove VXHS block deviceMarc-André Lureau2020-07-171-20/+2Star
| | | | | | | | | | | The vxhs code doesn't compile since v2.12.0. There's no point in fixing and then adding CI for a config that our users have demonstrated that they do not use; better to just remove it. Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20200711065926.2204721-1-marcandre.lureau@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* file-posix: Mitigate file fragmentation with extent size hintsKevin Wolf2020-07-141-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Especially when O_DIRECT is used with image files so that the page cache indirection can't cause a merge of allocating requests, the file will fragment on the file system layer, with a potentially very small fragment size (this depends on the requests the guest sent). On Linux, fragmentation can be reduced by setting an extent size hint when creating the file (at least on XFS, it can't be set any more after the first extent has been allocated), basically giving raw files a "cluster size" for allocation. This adds a create option to set the extent size hint, and changes the default from not setting a hint to setting it to 1 MB. The main reason why qcow2 defaults to smaller cluster sizes is that COW becomes more expensive, which is not an issue with raw files, so we can choose a larger size. The tradeoff here is only potentially wasted disk space. For qcow2 (or other image formats) over file-posix, the advantage should even be greater because they grow sequentially without leaving holes, so there won't be wasted space. Setting even larger extent size hints for such images may make sense. This can be done with the new option, but let's keep the default conservative for now. The effect is very visible with a test that intentionally creates a badly fragmented file with qemu-img bench (the time difference while creating the file is already remarkable) and then looks at the number of extents and the time a simple "qemu-img map" takes. Without an extent size hint: $ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0 $ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0 Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192) Run completed in 25.848 seconds. $ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096 Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192) Run completed in 19.616 seconds. $ filefrag ~/tmp/test.raw /home/kwolf/tmp/test.raw: 2000000 extents found $ time ./qemu-img map ~/tmp/test.raw Offset Length Mapped to File 0 0x1e8480000 0 /home/kwolf/tmp/test.raw real 0m1,279s user 0m0,043s sys 0m1,226s With the new default extent size hint of 1 MB: $ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576 $ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0 Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192) Run completed in 11.833 seconds. $ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096 Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192) Run completed in 10.155 seconds. $ filefrag ~/tmp/test.raw /home/kwolf/tmp/test.raw: 178 extents found $ time ./qemu-img map ~/tmp/test.raw Offset Length Mapped to File 0 0x1e8480000 0 /home/kwolf/tmp/test.raw real 0m0,061s user 0m0,040s sys 0m0,014s Signed-off-by: Kevin Wolf <kwolf@redhat.com> Message-Id: <20200707142329.48303-1-kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* block/qcow2: implement blockdev-amendMaxim Levitsky2020-07-061-1/+15
| | | | | | | | | | | Currently the implementation only supports amending the encryption options, unlike the qemu-img version Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200608094030.670121-14-mlevitsk@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* block/crypto: implement blockdev-amendMaxim Levitsky2020-07-061-1/+13
| | | | | | | | Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200608094030.670121-13-mlevitsk@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* block/core: add generic infrastructure for x-blockdev-amend qmp commandMaxim Levitsky2020-07-061-0/+42
| | | | | | | | | | | | | blockdev-amend will be used similiar to blockdev-create to allow on the fly changes of the structure of the format based block devices. Current plan is to first support encryption keyslot management for luks based formats (raw and embedded in qcow2) Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20200608094030.670121-12-mlevitsk@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* qcow2: Expose bitmaps' size during measureEric Blake2020-05-281-5/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional QMP field, present when measuring an existing image and output format that both support bitmaps. Update iotest 178 and 190 to updated output, as well as new coverage in 190 demonstrating non-zero values made possible with the recently-added qemu-img bitmap command (see 3b51ab4b). The new 'bitmaps size:' field is displayed automatically as part of 'qemu-img measure' any time it is present in QMP (that is, any time both the source image being measured and destination format support bitmaps, even if the measurement is 0 because there are no bitmaps present). If the field is absent, it means that no bitmaps can be copied (source, destination, or both lack bitmaps, including when measuring based on size rather than on a source image). This behavior is compatible with an upcoming patch adding 'qemu-img convert --bitmaps': that command will fail in the same situations where this patch omits the field. The addition of a new field demonstrates why we should always zero-initialize qapi C structs; while the qcow2 driver still fully populates all fields, the raw and crypto drivers had to be tweaked to avoid uninitialized data. Consideration was also given towards having a 'qemu-img measure --bitmaps' which errors out when bitmaps are not possible, and otherwise sums the bitmaps into the existing allocation totals rather than displaying as a separate field, as a potential convenience factor. But this was ultimately decided to be more complexity than necessary when the QMP interface was sufficient enough with bitmaps remaining a separate field. See also: https://bugzilla.redhat.com/1779904 Reported-by: Nir Soffer <nsoffer@redhat.com> Signed-off-by: Eric Blake <eblake@redhat.com> Message-Id: <20200521192137.1120211-3-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
* qcow2: add zstd cluster compressionDenis Plotnikov2020-05-131-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available. The performance test results: Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest. Image cluster size: 64K. Image on disk size: 2.2G The test was conducted with brd disk to reduce the influence of disk subsystem to the test results. The results is given in seconds. compress cmd: time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd] src.img [zlib|zstd]_compressed.img decompress cmd time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img compression decompression zlib zstd zlib zstd ------------------------------------------------------------ real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %) user 65.0 15.8 5.3 2.5 sys 3.3 0.2 2.0 2.0 Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com> QAPI part: Acked-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20200507082521.29210-4-dplotnikov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* qcow2: introduce compression type featureDenis Plotnikov2020-05-131-2/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The patch adds some preparation parts for incompatible compression type feature to qcow2 allowing the use different compression methods for image clusters (de)compressing. It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters. The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB. The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions. Adding of the compression type breaks a number of tests because now the compression type is reported on image creation and there are some changes in the qcow2 header in size and offsets. The tests are fixed in the following ways: * filter out compression_type for many tests * fix header size, feature table size and backing file offset affected tests: 031, 036, 061, 080 header_size +=8: 1 byte compression type 7 bytes padding feature_table += 48: incompatible feature compression type backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change) * add "compression type" for test output matching when it isn't filtered affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206, 242, 255, 274, 280 Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> QAPI part: Acked-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* qapi: Mark deprecated QMP parts with feature 'deprecated'Markus Armbruster2020-03-171-11/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add feature 'deprecated' to the deprecated QMP commands, so their deprecation becomes visible in output of query-qmp-schema. Looks like this: {"name": "query-cpus", "ret-type": "[164]", "meta-type": "command", "arg-type": "0", ---> "features": ["deprecated"]} Management applications could conceivably use this for static checking. The deprecated commands are change, cpu-add, migrate-set-cache-size, migrate_set_downtime, migrate_set_speed, query-cpus, query-events, query-migrate-cache-size. The deprecated command arguments are block-commit arguments @base and @top, and block_set_io_throttle, blockdev-change-medium, blockdev-close-tray, blockdev-open-tray, eject argument @device. The deprecated command results are query-cpus-fast result @arch, query-block result @dirty-bitmaps, query-named-block-nodes result @encryption_key_missing and result @dirty-bitmaps's member @status. Same for query-block result @inserted, which mirrors query-named-block-nodes. Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20200317115459.31821-27-armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>