In this bug: https://www.veripool.org/issues/1529-Verilator-Sefault-on-short-testcase
There was a request to file a bug with info about the fuzzing the produced issues 1529-1533 was run. This is that bug.
These were done with American Fuzzy Lop (http://lcamtuf.coredump.cx/afl/)
The attached script should be do this when run on a Debian or Ubuntu based machine.
This script assumes that it is run as root. This is only needed for some of the steps. In general, running programs that are expected to have unusual behavior as root is probably not a great idea - you may want to edit this. This should get things started though.
Note that the tool's idea of what's an "interesting crash" is not perfect. There are both cases where something looks fine to the tool but it actually produced the "buffer overflow" error, and also cases where what it classifies as a crash do not have any obvious problem. So I also have a script which I was using to rerun the cases that it found. This is in "actual_fail.py". This assumes a couple of things about paths that will need to be edited.
#2 Updated by Eric Rippey 20 days ago
As far as I know there isn't an option to keep the inputs ASCII.
There is an option to use a collection of interesting strings. That could be used to make the fuzzer know the whole set of valid keywords for Verilog and the set of valid symbols. I would expect that this would make things way more efficient. I've never used this option though and someone would have to write this list.
A different thing that can be done to speed up the fuzzer is to give it a larger corpus of examples to start with. In my example script, I created a single starting point, but to find the bugs that I filed I started with a set of around 1000 short files that I had around for other reasons. A good place to start might be to point it at the existing files in the test_regress/t directory.
You're right that two minutes is nowhere near how long things are expected to take. Assuming that you're seeing the number of paths found increase, it's not a bad idea to leave it to run overnight although I was seeing some results within the first half hour.
#3 Updated by Wilson Snyder 20 days ago
- Status changed from New to Feature
This is much appreciated.
1. First legal stuff, do you agree what you attach and future contributions are open source as described here? https://developercertificate.org/
How far do you want to take this? Not sure if you are willing to do more work, certainly would be helpful. Here's some suggestions based on your ideas:
2. Make nodist/* files we can put in the distro that run with "cwd" in the root of the verilator kit (e.g. cd <git area>; nodist/fuzzer). (Seems like just a few tweaks needed for this.)
3. Add corpus from test_regress?
4. Limit to ASCII if possible?
5. Add words based on tokens from verilog.l, maybe with the letters a-z for symbol use?
#4 Updated by Eric Rippey 13 days ago
The contributor agreement is fine.
At this point, I don't have a real bird's eye view of the how the project works to be able to how exactly things should be integrated. For example, where would you want this in the tree?
And where would this be expected to be run? Is there a set of platforms that I should make sure things work on? From reading the bug about using Travis CI, it seems like there isn't a set of dedicated machines anywhere to run things on.
Is there a document somewhere that lays out some development guidelines? For example, one of the scripts that I attached is Python and that might be the first piece of Python in the tree.
#5 Updated by Wilson Snyder 13 days ago
I'd suggest putting the files under the "nodist" directory, this is for things that are developer only and don't need to go into a tarball.
I personally ran the fuzzing from the $VERILATOR_ROOT directory (where the .git file is.)
I don't think we'd fuzz on Travis-CI, basically if if works on a standard Linux distro like Ubuntu 18.04 we're good.
There's some information in docs/CONTRIBUTING.adoc and docs/internals.adoc that might be of interest. (Make sure you pull first as these were recently cleaned up.)
Thanks for looking into this.
Also available in: Atom