Saving and restoring the state of a simulation

We're using Verilated blocks as part of a larger simulator and we'd like to be able to save and restore the state of the system. Looking at the C++ generated by Verilator, I think it should be possible by adding internal save/restore functions to each module that will work on the ports, signals and variables. Then in the top-level add public save/restore functions that will call the internal version on each module in the symbol table.

I was thinking it could use an ostream/istream to make it easy to write straight out to a file or a string to be written out later. We'd need some way of making sure the saved state was for an identical build, maybe just a hash of the source files. I don't see that this could work with --trace so the save/restore functions would only do anything if that was disabled. Perhaps add another command line option to enable this so we don't slow things down or increase code size for users who don't want it and make it incompatible with --trace.

Anything I've missed? Any comments?

Replies (3)

RE: Saving and restoring the state of a simulation - Added by Wilson Snyder over 2 years ago

The first (annoying perhaps) question is if you really need it. 95% of the times this has come up the save restore was to save having to do a slow "init" of a testbench every time. Instead of saving, come up with acceleration routines for example to directly load memory and registers. Although it requires more "human" work, it has the great advantage of accelerating even the "first" simulation, which is part of the most critical edit-compile-debug cycle.

Anyhow, assuming you really need it...

As you noted you won't easily be able to save/restore any open file handles, for tracing, or $fopens, though it is technically possible to save file pointers and resume them as append.

The simplest scheme though doesn't need any Verilator support (which is why it doesn't have any), simply create a core file and then load the core file en-mass. The nice thing about using core dumps is you can save-restore of the entire process, including your testbench classes. Roll your own or use one of the several "checkpoint" packages. You might not even need any code to do the save, just use "gcore".

If you do need save/restore functions though, use the level 2 I/O read() and write() calls. Streams are too slow for moving what might be 1GB of data for a large chip.

RE: Saving and restoring the state of a simulation - Added by Byron Bradley over 2 years ago

We do have a good reason for needing it, nothing to do being lazy :)

The core dumps isn't an option (although interesting generally). By the sounds of it adding this to mainline Verilator might do more harm than good but it would be a very good thing for us.

RE: Saving and restoring the state of a simulation - Added by Wilson Snyder over 2 years ago

Fair enough. I'm certainly willing to taking the changes back.

Some thoughts

A main decision you'll need to make is if you can save/restore to the same exact addresses or not. If not, you'll need to relink all of the data structures. (Hmm, it's probably easier to bring up a "new" model, then make sure to load over top of only the variable state, not any pointers.)

As for a dump format, I'd suggest making ELF files (man elf). They're nice because both lots of tools (like objdump) know how to read/write them and it's easy to write only a screen or so of code to read/write the format. I can send examples if you need them.

I'd suggest having Verilator make a MD5 sum of the design state, and save that also. On restore make sure it matches. I can make the checksum if you want, as I had uncommitted code about a year ago for an experimental feature that didn't pan out.

(1-3/3)