summaryrefslogtreecommitdiffstats
path: root/scripts
Commit message (Collapse)AuthorAgeFilesLines
...
| * | qapi: enable strict-optional checksJohn Snow2021-02-081-1/+0Star
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In the modules that we are checking so far, we can be stricter about the difference between Optional[T] and T types. Enable that check. Enabling it now will assist review on further typing and cleanup work. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-17-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
| * | qapi: type 'info' as Optional[QAPISourceInfo]John Snow2021-02-085-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For everything typed so far, type this parameter as Optional[QAPISourceInfo]. In the most generic case, QAPISchemaEntity's info field may be None to represent types that come from built-in definitions. Although some Entity types may not currently have any built-in definitions, it is not easily possible to constrain the type except on an ad-hoc basis using assertions. It's easier and simpler, then, to just say it's always an Optional type. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-16-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
| * | qapi/gen: Drop support for QAPIGen without a file nameMarkus Armbruster2021-02-081-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | The previous commit removed the only user of QAPIGen(None). Tighten the type hint. Signed-off-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-15-jsnow@redhat.com>
| * | qapi/commands: Simplify command registry generationMarkus Armbruster2021-02-081-27/+22Star
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | QAPISchemaGenCommandVisitor.visit_command() needs to generate the marshalling function into the current module, and also generate its registration into the ./init system module. The latter is done somewhat awkwardly: .__init__() creates a QAPIGenCCode that will not be written out, each .visit_command() adds its registration to it, and .visit_end() copies its contents into the ./init module it creates. Instead provide the means to temporarily switch to another module. Create the ./init module in .visit_begin(), and generate its initial part. Add registrations to it in .visit_command(). Finish it in .visit_end(). Signed-off-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-14-jsnow@redhat.com>
| * | qapi/gen: Support switching to another module temporarilyMarkus Armbruster2021-02-081-0/+7
| | | | | | | | | | | | | | | | | | | | | Signed-off-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-13-jsnow@redhat.com> [Commit message tweaked]
| * | qapi/gen: write _genc/_genh access shimsJohn Snow2021-02-081-5/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Many places assume they can access these fields without checking them first to ensure they are defined. Eliminating the _genc and _genh fields and replacing them with functional properties that check for correct state can ease the typing overhead by eliminating the Optional[T] return type. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-12-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
| * | qapi: centralize the built-in module name definitionJohn Snow2021-02-081-5/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | Use a constant to make it obvious we're referring to a very specific thing. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-11-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
| * | qapi/gen: Combine ._add_[user|system]_moduleMarkus Armbruster2021-02-083-14/+7Star
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With callers to _add_system_module now explicitly using the './' prefix to indicate a system module, there is no longer any reason to have separate interfaces for adding system vs user modules; use a unified interface that differentiates based on the name. Signed-off-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-10-jsnow@redhat.com>
| * | qapi: use './builtin' as the built-in module nameJohn Snow2021-02-082-19/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Use './builtin' as the built-in module name instead of None. Clarify the typing that this is now always a string. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-9-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
| * | qapi: use explicitly internal module namesJohn Snow2021-02-083-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | QAPISchemaModularCVisitor._add_system_module() prefixes './' to its name argument to make it a module name. Pass the module name instead. This will allow us to coalesce the methods to add modules later on. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-8-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Commit message reworded] Signed-off-by: Markus Armbruster <armbru@redhat.com>
| * | qapi/gen: Replace ._begin_system_module()Markus Armbruster2021-02-083-6/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | QAPISchemaModularCVisitor._begin_system_module() is actually just for the builtin module. Rename it to ._begin_builtin_module() and drop its useless @name parameter. Clarify conditionals in visit_module to make this clear. Signed-off-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-7-jsnow@redhat.com>
| * | qapi: centralize is_[user|system|builtin]_module methodsJohn Snow2021-02-082-16/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Define what a module is and define what kind of a module it is once and for all, in one place. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-6-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
| * | qapi/gen: inline _wrap_ifcond into end_if()John Snow2021-02-081-5/+2Star
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We assert _start_if is not None in end_if, but that's opaque to mypy. By inlining _wrap_ifcond, that constraint becomes provable to mypy. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-5-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
| * | qapi/main: handle theoretical None-return from re.match()John Snow2021-02-081-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Mypy cannot understand that this match can never be None, so help it along. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-4-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
| * | qapi/events: fix visit_event typingJohn Snow2021-02-081-5/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | Actually, the arg_type can indeed be Optional. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-3-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
| * | qapi/commands: assert arg_type is not NoneJohn Snow2021-02-081-3/+6
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When boxed is True, expr.py asserts that we must have arguments. Ultimately, this should mean that if boxed is True that arg_type should be defined. Mypy cannot infer this, and does not support 'stateful' type inference, e.g.: ``` if x: assert y is not None ... if x: y.etc() ``` does not work, because mypy does not statefully remember the conditional assertion in the second block. Help mypy out by creating a new local that it can track more easily. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-2-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
* / scripts/mtest2make.py: export all-%s-targets variable and use itAlex Bennée2021-02-081-0/+1
|/ | | | | | | | | | | | | There are some places where the conditional makefile support is the simplest solution. Now we don't expose CONFIG_TCG as a variable create a new one that can be checked for the check-help output. As check-tcg is a PHONY target we re-use check-softfloat to gate that as well. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20210202134001.25738-13-alex.bennee@linaro.org>
* simpletrace: build() missing 2 required positional argumentsVolker Rümelin2021-02-011-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | Commit 4e66c9ef64 "tracetool: add input filename and line number to Event" forgot to add a line number and a filename argument at one build method call site. Traceback (most recent call last): File "./scripts/simpletrace.py", line 261, in <module> run(Formatter()) File "./scripts/simpletrace.py", line 236, in run process(events, sys.argv[2], analyzer, read_header=read_header) File "./scripts/simpletrace.py", line 177, in process dropped_event = Event.build("Dropped_Event(uint64_t num_events_dropped)") TypeError: build() missing 2 required positional arguments: 'lineno' and 'filename' Add the missing arguments. Signed-off-by: Volker Rümelin <vr_qemu@t-online.de> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-id: 20210131173415.3392-1-vr_qemu@t-online.de Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
* trace: make the 'log' backend timestamp configurableStefan Hajnoczi2021-02-011-6/+13
| | | | | | | | | | | | | | Timestamps in tracing output can be distracting. Make it possible to control tid/timestamp printing with -msg timestamp=on|off. The default is no tid/timestamps. Previously they were always printed. Suggested-by: BALATON Zoltan <balaton@eik.bme.hu> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: BALATON Zoltan <balaton@eik.bme.hu> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-id: 20210125113507.224287-3-stefanha@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
* tracetool: also strip %l and %ll from systemtap format stringsDaniel P. Berrangé2021-02-011-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | All variables are 64-bit and so %l / %ll are not required, and the latter is actually invalid: $ sudo stap -e 'probe begin{printf ("BEGIN")}' -I . parse error: invalid or missing conversion specifier saw: operator ',' at ./qemu-system-x86_64-log.stp:15118:101 source: printf("%d@%d vhost_vdpa_set_log_base dev: %p base: 0x%x size: %llu refcnt: %d fd: %d log: %p\n", pid(), gettimeofday_ns(), dev, base, size, refcnt, fd, log) ^ Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Laurent Vivier <lvivier@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Laurent Vivier <lvivier@redhat.com> Message-id: 20210106130239.1004729-1-berrange@redhat.com [Fixed "simiarly" typo found by Laurent Vivier <lvivier@redhat.com> --Stefan] Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
* tracetool: fix "PRI" macro decodingLaurent Vivier2021-02-011-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | macro is not reset after use, so the format decoded is always the one of the first "PRI" in the format string. For instance: vhost_vdpa_set_config(void *dev, uint32_t offset, uint32_t size, \ uint32_t flags) "dev: %p offset: %"PRIu32" \ size: %"PRIu32" flags: 0x%"PRIx32 generates: printf("%d@%d vhost_vdpa_set_config dev: %p offset: %u size: %u \ flags: 0x%u\n", pid(), gettimeofday_ns(), dev, offset, \ size, flags) for the "flags" parameter, we can see a "0x%u" rather than a "0x%x" because the first macro was "PRIu32" (for offset). In the loop, macro becomes "PRIu32PRIu32PRIx32", and c_macro_to_format() returns always macro[3] ('u' in this case). This patch resets macro after the format has been decoded. Signed-off-by: Laurent Vivier <lvivier@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-id: 20210105191721.120463-3-lvivier@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
* Merge remote-tracking branch ↵Peter Maydell2021-01-291-8/+26
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 'remotes/berrange-gitlab/tags/misc-fixes-pull-request' into staging * Replace --enable/disable-git-update with --with-git-submodules to allow improved control over use of git submodules * Deprecate the -enable-fips option * Ensure docs use prefer format for bool options * Clarify platform support rules * Misc fixes to keymap conversions * Fix misc problems on macOS # gpg: Signature made Fri 29 Jan 2021 17:10:13 GMT # gpg: using RSA key DAF3A6FDB26B62912D0E8E3FBE86EBB415104FDF # gpg: Good signature from "Daniel P. Berrange <dan@berrange.com>" [full] # gpg: aka "Daniel P. Berrange <berrange@redhat.com>" [full] # Primary key fingerprint: DAF3 A6FD B26B 6291 2D0E 8E3F BE86 EBB4 1510 4FDF * remotes/berrange-gitlab/tags/misc-fixes-pull-request: tests: Replace deprecated ASN1 code tests: Fix runtime error in test-authz-pam ui: update keycodemapdb submodule commit crypto: Add spaces around operator configure: replace --enable/disable-git-update with --with-git-submodules docs: fix missing backslash in certtool shell example docs: simplify and clarify the platform support rules Prefer 'on' | 'off' over 'yes' | 'no' for bool options os: deprecate the -enable-fips option and QEMU's FIPS enforcement crypto: Fix memory leaks in set_loaded for tls-* crypto: Forbid broken unloading of secrets crypto: Move USER_CREATABLE to secret_common base class crypto: Fix some code style problems, add spaces around operator Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
| * configure: replace --enable/disable-git-update with --with-git-submodulesDan Streetman2021-01-291-8/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Replace the --enable-git-update and --disable-git-update configure params with the param --with-git-submodules=(update|validate|ignore) to allow 3 options for building from a git repo. This is needed because downstream packagers, e.g. Debian, Ubuntu, etc, also keep the source code in git, but do not want to enable the 'git_update' mode; with the current code, that's not possible even if the downstream package specifies --disable-git-update. The previous parameters are deprecated but still available; the --enable-git-update parameter maps to --with-git-submodules=update and --disable-git-update parameter maps to --with-git-submodules=validate. The configure script behavior is slightly modified, where previously the dtc, capstone, and slirp submodules were not validated when --disable-git-update was specified (but were updated with git-update enabled), now they are validated when using --with-git-submodules=validate and are only ignored when using --with-git-submodules=ignore. Signed-off-by: Dan Streetman <ddstreet@canonical.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
* | hvf: Add hypervisor entitlement to output binariesAlexander Graf2021-01-291-0/+13
|/ | | | | | | | | | | In macOS 11, QEMU only gets access to Hypervisor.framework if it has the respective entitlement. Add an entitlement template and automatically self sign and apply the entitlement in the build. Signed-off-by: Alexander Graf <agraf@csgraf.de> Reviewed-by: Roman Bolshakov <r.bolshakov@yadro.com> Tested-by: Roman Bolshakov <r.bolshakov@yadro.com> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
* simplebench: add bench-backup.pyVladimir Sementsov-Ogievskiy2021-01-261-0/+167
| | | | | | | | | Add script to benchmark new backup architecture. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20210116214705.822267-24-vsementsov@virtuozzo.com> [mreitz: s/not unsupported/not supported/] Signed-off-by: Max Reitz <mreitz@redhat.com>
* simplebench: bench_block_job: add cmd_options argumentVladimir Sementsov-Ogievskiy2021-01-262-5/+8
| | | | | | | | | Add argument to allow additional block-job options. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-23-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* simplebench/bench_block_job: use correct shebang line with python3Vladimir Sementsov-Ogievskiy2021-01-261-1/+1
| | | | | | | Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-22-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* update-linux-headers: Include const.hEric Farman2021-01-211-1/+4
| | | | | | | | | | | | | Kernel commit a85cbe6159ff ("uapi: move constants from <linux/kernel.h> to <linux/const.h>") breaks our script because of the unrecognized include. Let's add that to our processing. Signed-off-by: Eric Farman <farman@linux.ibm.com> Reviewed-by: Peter Xu <peterx@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210104202057.48048-2-farman@linux.ibm.com> Signed-off-by: Cornelia Huck <cohuck@redhat.com>
* scripts/checkpatch.pl: fix git-show invocation to include diffstatAlex Bennée2021-01-181-1/+1
| | | | | | | | | | | | Without this checkpatch keeps complaining about new/changed files even when MAINTAINERS has been updated. Normal invocations of checkpatch on patch files rather than commit IDs are unaffected. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210114165730.31607-13-alex.bennee@linaro.org>
* decodetree: Open files with encoding='utf-8'Philippe Mathieu-Daudé2021-01-131-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When decodetree.py was added in commit 568ae7efae7, QEMU was using Python 2 which happily reads UTF-8 files in text mode. Python 3 requires either UTF-8 locale or an explicit encoding passed to open(). Now that Python 3 is required, explicit UTF-8 encoding for decodetree source files. To avoid further problems with the user locale, also explicit UTF-8 encoding for the generated C files. Explicit both input/output are plain text by using the 't' mode. This fixes: $ /usr/bin/python3 scripts/decodetree.py test.decode Traceback (most recent call last): File "scripts/decodetree.py", line 1397, in <module> main() File "scripts/decodetree.py", line 1308, in main parse_file(f, toppat) File "scripts/decodetree.py", line 994, in parse_file for line in f: File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 80: ordinal not in range(128) Reported-by: Peter Maydell <peter.maydell@linaro.org> Suggested-by: Yonggang Luo <luoyonggang@gmail.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20210110000240.761122-1-f4bug@amsat.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* scripts/gdb: implement 'qemu bt'Maxim Levitsky2021-01-122-1/+28
| | | | | | | | | | This script first runs the regular gdb's 'bt' command, and then if we are in a coroutine it prints the coroutines backtraces in the order in which they were called. Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> Message-Id: <20201217155436.927320-3-mlevitsk@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* scripts/gdb: fix 'qemu coroutine' when users selects a non topmost stack frameMaxim Levitsky2021-01-121-0/+7
| | | | | | | | | | | | | | | | | | | | | The code that dumps the stack frame works like that: * save current registers * overwrite current registers (including rip/rsp) with coroutine snapshot in the jmpbuf * print backtrace * restore the saved registers. If the user has currently selected a non topmost stack frame in gdb, the above code will still restore the selected frame registers, but the gdb will then lose the selected frame index, which makes it impossible to switch back to frame 0, to continue debugging the executable. Therefore switch temporarily to the topmost frame of the stack for the above code. Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> Message-Id: <20201217155436.927320-2-mlevitsk@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* fuzz: heuristic split write based on past IOsQiuhao Li2021-01-111-0/+56
| | | | | | | | | | | If previous write commands write the same length of data with the same step, we view it as a hint. Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com> Reviewed-by: Alexander Bulekov <alxndr@bu.edu> Tested-by: Alexander Bulekov <alxndr@bu.edu> Message-Id: <SYCPR01MB3502480AD07811A6A49B8FEAFCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* fuzz: add minimization optionsQiuhao Li2021-01-111-4/+26
| | | | | | | | | | | -M1: remove IO commands iteratively -M2: try setting bits in operand of write/out to zero Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com> Reviewed-by: Alexander Bulekov <alxndr@bu.edu> Tested-by: Alexander Bulekov <alxndr@bu.edu> Message-Id: <SYCPR01MB350204C52E7A39E6B0EEC870FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* fuzz: set bits in operand of write/out to zeroQiuhao Li2021-01-111-0/+39
| | | | | | | | | | | | | | Simplifying the crash cases by opportunistically setting bits in operands of out/write to zero may help to debug, since usually bit one means turn on or trigger a function while zero is the default turn-off setting. Tested bug https://bugs.launchpad.net/qemu/+bug/1908062 Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com> Reviewed-by: Alexander Bulekov <alxndr@bu.edu> Tested-by: Alexander Bulekov <alxndr@bu.edu> Message-Id: <SYCPR01MB3502C84B6346A3E3DE708C7BFCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* fuzz: remove IO commands iterativelyQiuhao Li2021-01-111-15/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Now we use a one-time scan and remove strategy in the minimizer, which is not suitable for timing dependent instructions. For example, instruction A will indicate an address where the config chunk locates, and instruction B will make the configuration active. If we have the following instruction sequence: ... A1 B1 A2 B2 ... A2 and B2 are the actual instructions that trigger the bug. If we scan from top to bottom, after we remove A1, the behavior of B1 might be unknowable, including not to crash the program. But we will successfully remove B1 later cause A2 and B2 will crash the process anyway: ... A1 A2 B2 ... Now one more trimming will remove A1. In the perfect case, we would need to be able to remove A and B (or C!) at the same time. But for now, let's just add a loop around the minimizer. Since we only remove instructions, this iterative algorithm is converging. Tested with Bug 1908062. Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com> Reviewed-by: Alexander Bulekov <alxndr@bu.edu> Tested-by: Alexander Bulekov <alxndr@bu.edu> Message-Id: <SYCPR01MB350263004448040ACCB9A9F1FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* fuzz: split write operand using binary approachQiuhao Li2021-01-111-9/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, we split the write commands' data from the middle. If it does not work, try to move the pivot left by one byte and retry until there is no space. But, this method has two flaws: 1. It may fail to trim all unnecessary bytes on the right side. For example, there is an IO write command: write addr uuxxxxuu u is the unnecessary byte for the crash. Unlike ram write commands, in most case, a split IO write won't trigger the same crash, So if we split from the middle, we will get: write addr uu (will be removed in next round) write addr xxxxuu For xxxxuu, since split it from the middle and retry to the leftmost byte won't get the same crash, we will be stopped from removing the last two bytes. 2. The algorithm complexity is O(n) since we move the pivot byte by byte. To solve the first issue, we can try a symmetrical position on the right if we fail on the left. As for the second issue, instead moving by one byte, we can approach the boundary exponentially, achieving O(log(n)). Give an example: xxxxuu len=6 + | + xxx,xuu 6/2=3 fail + +--------------+-------------+ | | + + xx,xxuu 6/2^2=1 fail xxxxu,u 6-1=5 success + + +------------------+----+ | | | +-------------+ u removed + + xx,xxu 5/2=2 fail xxxx,u 6-2=4 success + | +-----------+ u removed In some rare cases, this algorithm will fail to trim all unnecessary bytes: xxxxxxxxxuxxxxxx xxxxxxxx-xuxxxxxx Fail xxxx-xxxxxuxxxxxx Fail xxxxxxxxxuxx-xxxx Fail ... I think the trade-off is worth it. Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com> Reviewed-by: Alexander Bulekov <alxndr@bu.edu> Tested-by: Alexander Bulekov <alxndr@bu.edu> Message-Id: <SYCPR01MB3502D26F1BEB680CBBC169E5FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* fuzz: double the IOs to remove for every loopQiuhao Li2021-01-111-12/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of removing IO instructions one by one, we can try deleting multiple instructions at once. According to the locality of reference, we double the number of instructions to remove for the next round and recover it to one once we fail. This patch is usually significant for large input. Test with quadrupled trace input at: https://bugs.launchpad.net/qemu/+bug/1890333/comments/1 Patched 1/6 version: real 0m45.904s user 0m16.874s sys 0m10.042s Refined version: real 0m11.412s user 0m6.888s sys 0m3.325s Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com> Reviewed-by: Alexander Bulekov <alxndr@bu.edu> Tested-by: Alexander Bulekov <alxndr@bu.edu> Message-Id: <SYCPR01MB350280A67BB55C3FADF173E3FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* fuzz: accelerate non-crash detectionQiuhao Li2021-01-111-13/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We spend much time waiting for the timeout program during the minimization process until it passes a time limit. This patch hacks the CLOSED (indicates the redirection file closed) notification in QTest's output if it doesn't crash. Test with quadrupled trace input at: https://bugs.launchpad.net/qemu/+bug/1890333/comments/1 Original version: real 1m37.246s user 0m13.069s sys 0m8.399s Refined version: real 0m45.904s user 0m16.874s sys 0m10.042s Note: Sometimes the mutated or the same trace may trigger a different crash summary (second-to-last line) but indicates the same bug. For example, Bug 1910826 [1], which will trigger a stack overflow, may output summaries like: SUMMARY: AddressSanitizer: stack-overflow /home/qiuhao/hack/qemu/build/../softmmu/physmem.c:488 in flatview_do_translate or SUMMARY: AddressSanitizer: stack-overflow (/home/qiuhao/hack/qemu/build/qemu-system-i386+0x27ca049) in __asan_memcpy Etc. If we use the whole summary line as the token, we may be prevented from further minimization. So in this patch, we only use the first three words which indicate the type of crash: SUMMARY: AddressSanitizer: stack-overflow [1] https://bugs.launchpad.net/qemu/+bug/1910826 Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com> Reviewed-by: Alexander Bulekov <alxndr@bu.edu> Tested-by: Alexander Bulekov <alxndr@bu.edu> Message-Id: <SYCPR01MB350251DC04003450348FAF68FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* scripts/coccinelle: New script to remove unnecessary timer_del() callsPeter Maydell2021-01-081-0/+18
| | | | | | | | | | | | | | | | | Now that timer_free() implicitly calls timer_del(), sequences timer_del(mytimer); timer_free(mytimer); can be simplified to just timer_free(mytimer); Add a Coccinelle script to do this transformation. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Acked-by: Paolo Bonzini <pbonzini@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20201215154107.3255-3-peter.maydell@linaro.org
* tracetool: show trace-events filename/lineno in fmt string errorsStefan Hajnoczi2021-01-043-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | The compiler encounters trace event format strings in generated code. Format strings are error-prone and therefore clear compiler errors are important. Use the #line directive to show the trace-events filename and line number in format string errors: https://gcc.gnu.org/onlinedocs/gcc-10.2.0/cpp/Line-Control.html For example, if the cpu_in trace event's %u is changed to %p the following error is reported: trace-events:29:18: error: format ‘%p’ expects argument of type ‘void *’, but argument 7 has type ‘unsigned int’ [-Werror=format=] Line 29 in trace-events is where cpu_in is defined. This works for any trace-events file in the QEMU source tree and the correct path is displayed. Unfortunately there does not seem to be a way to set the column, so "18" is not the right character on that line. Suggested-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-Id: <20200827142915.108730-5-stefanha@redhat.com>
* tracetool: add input filename and line number to EventStefan Hajnoczi2021-01-041-5/+22
| | | | | | | | | | | Store the input filename and line number in Event. A later patch will use this to improve error messages. Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-Id: <20200827142915.108730-4-stefanha@redhat.com>
* tracetool: add out_lineno and out_next_lineno to out()Stefan Hajnoczi2021-01-041-1/+9
| | | | | | | | | | | Make the output file line number and next line number available to out(). A later patch will use this to improve error messages. Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-Id: <20200827142915.108730-3-stefanha@redhat.com>
* tracetool: add output filename command-line argumentStefan Hajnoczi2021-01-042-7/+23
| | | | | | | | | | | | | | The tracetool.py script writes to stdout. This means the output filename is not available to the script. Add the output filename to the command-line so that the script has access to the filename. This also simplifies the tracetool.py invocation. It's no longer necessary to use meson's custom_build(capture : true) to save output. Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-Id: <20200827142915.108730-2-stefanha@redhat.com>
* scripts/simplebench: add bench_prealloc.pyVladimir Sementsov-Ogievskiy2020-12-181-0/+132
| | | | | | | | | | | | | | | | | Benchmark for new preallocate filter. Example usage: ./bench_prealloc.py ../../build/qemu-img \ ssd-ext4:/path/to/mount/point \ ssd-xfs:/path2 hdd-ext4:/path3 hdd-xfs:/path4 The benchmark shows performance improvement (or degradation) when use new preallocate filter with qcow2 image. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20201021145859.11201-22-vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* simplebench/results_to_text: make executableVladimir Sementsov-Ogievskiy2020-12-181-0/+14
| | | | | | | | | Make results_to_text a tool to dump results saved in JSON file. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20201021145859.11201-21-vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* simplebench/results_to_text: add difference line to the tableVladimir Sementsov-Ogievskiy2020-12-181-7/+60
| | | | | | | | | | | Performance improvements / degradations are usually discussed in percentage. Let's make the script calculate it for us. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20201021145859.11201-20-vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> [mreitz: 'seconds' instead of 'secs'] Signed-off-by: Max Reitz <mreitz@redhat.com>
* simplebench/results_to_text: improve view of the tableVladimir Sementsov-Ogievskiy2020-12-181-1/+12
| | | | | | | | | Move to generic format for floats and percentage for error. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20201021145859.11201-19-vsementsov@virtuozzo.com> Acked-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* simplebench: move results_to_text() into separate fileVladimir Sementsov-Ogievskiy2020-12-184-33/+52
| | | | | | | | | | Let's keep view part in separate: this way it's better to improve it in the following commits. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20201021145859.11201-18-vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* simplebench: rename ascii() to results_to_text()Vladimir Sementsov-Ogievskiy2020-12-183-7/+7
| | | | | | | | | | Next patch will use utf8 plus-minus symbol, let's use more generic (and more readable) name. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20201021145859.11201-17-vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
generated by cgit v1.2.3-55-g7522 (git 2.39.0) at 2024-12-18 09:16:35 +0100