Make verilator_ext_tests head-to-head
This is still a WIP, but I think it warrants some discussion before I trudge ahead much further: https://github.com/toddstrader/verilator_ext_tests/tree/head-to-head
I'm proposing that we have Travis always pull the latest master from every submodule (including Verilator, which I added). This way we don't have to keep bumping these submodules by hand. Right now, I'm just running Travis on my branch manually, or by pushing verilator_ext_tests. If we go with head-to-head, we can initially just turn on daily cron builds.
There's also some fanciness in Travis about public/private keys, encrypted YAML and protected environment variables. This supposedly allows you to "safely" kick off Travis builds via webhooks. I'm wondering if we could grant these projects we're building with these privileges so that they could kick off a verilator_ext_tests build when they run their own CI. This would just grant them the ability to control Travis runs and not permissions to GitHub. All of this is to say, I would need to do more research on this bit.
You'll notice that Travis is currently failing the SweRV test: https://travis-ci.com/toddstrader/verilator_ext_tests/builds/120918474
This is actually a fortunate coincidence because the SweRV test passes if you run with the Verilator rev that is committed via the submodule (just don't update with --remote). But if you do update to HEAD of verilator, it breaks. So . . . heads up SweRV people (I'll post there).
I also learned how to cache the full Verilator build with Travis build stages. This has potential benefits for the Verilator proper CI. I'll have to circle back there. However, the solution I've landed on is a bit cumbersome. Essentially, I'd like to install Verilator with installation option #4 (from README.pod). However, this involves unsetting VERILATOR_ROOT which breaks driver.pl which these tests depend on. I'm wondering if VERILATOR_ROOT can be deprecated, but that's a topic for another thread.
What are your thoughts on this approach? It will obviously break more often. But I think that's what we want. If we have to manually bump submodule versions, I'm not sure how much more beneficial verilator_ext_tests is than these projects' own CI environments.
#1 Updated by Wilson Snyder about 2 months ago
- Status changed from New to Assigned
I agree this is a good approach. Note Verilator, along with SwerRV is part of Chips Alliance, so if you want you can ask for commit privileges to it, and/or we can move the ext tests to the CHIPS Alliance repo if that helps in some way (e.g. they might pay for more Travis services).
#3 Updated by Todd Strader 30 days ago
There's more to do here, but I'm focused on the DPI protected modules for now. But I believe this should be land-able: https://github.com/toddstrader/verilator_ext_tests/tree/head-to-head
If we turn on daily cron jobs, we'll get a head-to-head check of all these tests and Verilator every day.
Also, the build has gone red since I looked at this last: https://travis-ci.com/toddstrader/verilator_ext_tests/builds/122567524
This looks to be a legitimate SweRV issue this time. I suggest we add a requirement to the README that external projects have their own CI using whatever their fixed version of Verilator is. We can grandfather in the current two projects and politely ask them to set up their own CI.
#4 Updated by Wilson Snyder 25 days ago
- Category set to Tests
These changes look fine.
Personally I'd like to be able to run the tests pointing at a separate verilator tree (I am developing at that moment) without picking up the external for verilator. Do you know if there's a way we can do that? Or maybe I just live with the external fetch and set VERILATOR_ROOT to my local area?
#5 Updated by Todd Strader 24 days ago
That's what I've been doing. I just added ci/update_submodules.sh which does the remote submodule update for all submodules but excludes the verilator submodule if $VERILATOR_ROOT is set to something other than the in-tree copy. Does this work for your workflow?
Also available in: Atom