Project

General

Profile

[logo] 
 
Home
News
Activity
About/Contact
Major Tools
  Dinotrace
  Verilator
  Verilog-mode
  Verilog-Perl
Other Tools
  IPC::Locker
  Parallel::Forker
  Voneline
General Info
  Papers

Building Verilator with Bazel

Added by Kevin Kiningham over 1 year ago

I use the Bazel build system a lot for my own projects and have created a fork of Verilator that supports Bazel. It's available on github at https://github.com/kkiningh/rules_verilator/tree/bazel. Bazel is a fork of Blaze, the build system used internally at Google. It also supports many advanced features not available in make such as remote caching, distributed building, integrated testing, etc. I also have rules for interfacing Verilog and C++ using Verilator and can add them to the fork if there's interest.

I was able to get it working without any code changes to Verilator itself, despite a few rough edges. However, there is one particular change that would be nice to have in Verilator, and I was hoping to see if there was any possibility that it could be implemented.

In Bazel, all outputs of a rule must be known statically. However, Verilator can generate any number of files depending on things like inlining, etc. Would it be possible to add a flag that forces Verilator to generate a predictable set of source files given the same flags and modules as input? It's fine if the set is large or some of those files are empty (for example if module A gets inlined, it's ok if it's corresponding .cpp and .h are empty, as long as they exist) but having at least a stable set of files would be a huge improvement.

Thanks!


Replies (23)

RE: Building Verilator with Bazel - Added by Wilson Snyder over 1 year ago

Verilator could solve the problem for one of your modules by having say always 50 output files, but as soon as someone edits the verilog to add/remove an instantiated module your build system would break (as both input and output change), which seems a nasty thing to do to your users. Or maybe I'm missing something in your request.

Perhaps Bazel is an exception, but all build systems I've seen usually have some way around this, perhaps with some hack like recursion. Have you asked the Bazel team how to handle languages where the language determines the input and output filenames?

RE: Building Verilator with Bazel - Added by Kevin Kiningham over 1 year ago

Verilator could solve the problem for one of your modules by having say always 50 output files, but as soon as someone edits the verilog to add/remove an instantiated module your build system would break (as both input and output change), which seems a nasty thing to do to your users.

The way I have it setup right now is that for every Verilog file you're required to list all declared modules/interfaces/etc plus any dependencies in a corresponding BUILD file. Then, when building, Bazel looks up all the transitive dependencies of the file containing the top module and calls Verilator with that set of files. Ideally, since Bazel knows all the modules/packages/etc declared in a fileset, it should be possible to derive a consistent set of output files.

To be clear, what I'm asking for is that if Verilator is given the same set of input files, and those files declare the same set of modules/interfaces/etc, then Verilator will produce the same set of output files. If a user adds a new module or introduces a new dependency between modules, it's okay for the output set to change since the user will also have to update the BUILD file, which will inform Bazel. Additionally, it's okay if some of the files are empty or stubs, as long as the set is consistent.

Have you asked the Bazel team how to handle languages where the language determines the input and output filenames?

It depends. Bazel isn't like other build systems in the sense that it has a hard requirement that it knows all inputs for a rule and enforces that requirement by sandboxing the compiler. For a language like C++ (where a file depends on header files included by the preprocessor) the user has to tell Bazel what files the C++ file is including. This might be by explicitly writing out all the names of header files, or depending on a "library" rule that declares the header files as outputs (note that "output" doesn't necessarily mean generated, it just means that if rule B depends on rule A, rule B can read all files marked as rule A's "output"). Bazel also provides macros and ways to gather transitive dependencies, so listing out dependencies like this is usually a similar amount of effort as defining dependencies in make.

If the set of input files is completely arbitrary, you need to write a wrapper script that takes in some defined set of files and then calls the actual tool. For example, the wrapper might untar a directory containing all possible input files and the rule would just depend on the tar file. However, it's pretty unusual for a tool to require truly arbitrary inputs in practice, so I haven't actually seen a rule that does this and the builtin rules don't support it.

RE: Building Verilator with Bazel - Added by Wilson Snyder over 1 year ago

Interesting, don't think I'd personally like our team to have to edit dependencies when they edit Verilog, but each your own.

To be clear, what I'm asking for is that if Verilator is given the same set of input files, and those files declare the same set of modules/interfaces/etc, then Verilator will produce the same set of output files. If a user adds a new module or introduces a new dependency between modules, it's okay for the output set to change since the user will also have to update the BUILD file, which will inform Bazel.

Do you really want the user to have to figure out which outputs?

I'm thinking name all of the outputs with some common prefix and always make N files, if all N aren't needed make them empty. If N+1 are needed, error out saying to increase N?

RE: Building Verilator with Bazel - Added by Kevin Kiningham over 1 year ago

Interesting, don't think I'd personally like our team to have to edit dependencies when they edit Verilog, but each your own.

Edit the BUILD file, and it's only needed when you add a new dependency or a new module. I was skeptical too when I started, but it's actually very little work (basically add a single word) and you get a lot of benefits from using Bazel.

Do you really want the user to have to figure out which outputs?

Ideally, Bazel should be able to figure out the names with no user intervention.

I'm thinking name all of the outputs with some common prefix and always make N files, if all N aren't needed make them empty. If N+1 are needed, error out saying to increase N?

Right now, the set of output files seems to be something like:

    {prefix}.cpp
    {prefix}.h
    {prefix}__Syms.cpp
    {prefix}__Slow.cpp
    {prefix}__Inlines.h
    {prefix}{module}.cpp
    {prefix}{module}.h

but sometimes it doesn't produce files for some modules depending on the optimization mode. If the only change was that it always created files following this format for all modules/interfaces/etc found, regardless of optimization mode, that would be perfect.

RE: Building Verilator with Bazel - Added by Wilson Snyder over 1 year ago

If I was using C etc, I see how this fits in well, but I still think it's counter to how most Verilog RTL+Verif engineers expect builds to work (e.g. they expect they can change a parameter and have the dependencies change to reflect that new parameter value.) Anyhow.

Right now Verilator really names the files sometimes also with a number (--output-split)

{prefix}{module}{__n}.cpp
{prefix}{module}{__n}.h

What I was suggesting is to no longer put in module name and use

{prefix}{__n}.cpp
{prefix}{__n}.h

Where if you do, say --output-numerated 50, Verilator will make

{prefix}__0.cpp
{prefix}__0.h
to..... __49.cpp
to..... __49.h

If verilator ended up needing only 30 outputs the remaining ones would be empty. If 51 were needed, error.

If this is suitable perhaps you would be willing to submit a patch? Should be straight-forward, I'll provide some hints.

RE: Building Verilator with Bazel - Added by Kevin Kiningham over 1 year ago

e.g. they expect they can change a parameter and have the dependencies change to reflect that new parameter value.

Not sure I understand - are you saying that if module A conditionally instantiates module B based on a parameter value, you expect the build system will conditionally include the file that contains B when it calls Verilator? If that's the case, Bazel supports this, but you need to tell Bazel what the conditions are (as I would imagine you would need to in any other build system). Not entirely sure I understand the situation you're describing though.

Where if you do, say --output-numerated 50, Verilator will make

How would header files work? E.g. if my testbench relies on the header for a module top, right now I include "{prefix}top.h". If the module name was not included, what do I include?

If this is suitable perhaps you would be willing to submit a patch? Should be straight-forward, I'll provide some hints.

I can try! I'm not super familiar with the codebase, but I'd be happy to take a shot if you have some hints.

RE: Building Verilator with Bazel - Added by Wilson Snyder over 1 year ago

e.g. they expect they can change a parameter and have the dependencies change to reflect that new parameter value.

Not sure I understand - are you saying that if module A conditionally instantiates module B based on a parameter value, you expect the build system will conditionally include the file that contains B when it calls Verilator? If that's the case, Bazel supports this, but you need to tell Bazel what the conditions are (as I would imagine you would need to in any other build system). Not entirely sure I understand the situation you're describing though.

Yes. The build systems I've used to date always would automatically use the dependencies Verilator/the simulator found. I understand you could express them to the build system, but this is duplicating information already expressed in Verilog and I hate duplication perhaps to a fault.

How would header files work? E.g. if my testbench relies on the header for a module top, right now I include "{prefix}top.h". If the module name was not included, what do I include?

Actually "top" is part of prefix, so you're loading {prefix}.h. However your primary point is right, we should leave the {prefix}.h file alone, and rename most the rest.

{prefix}.cpp                        // Rename
{prefix}.h                          // Keep
{prefix}{each_verilog_module}.cpp   // Rename
{prefix}{each_verilog_module}.h     // Rename
{prefix}__Dpi.h                     // Keep
{prefix}__Inlines.h                 // Keep
{prefix}__Slow.cpp                  // Rename with __Slow
{prefix}__Syms.cpp                  // Rename
{prefix}__Syms.h                    // Keep
{prefix}__Trace.cpp                 // Rename

Making this list I realize we should also make 50 Slow files.

I can try! I'm not super familiar with the codebase, but I'd be happy to take a shot if you have some hints.

1. Make a test case in test_regress (which will fail until done), see internals.txt, it can be just a hello-world, it would set say --output-numerate 10, then check for existence of the numbered files and that file __10 doesn't exist (off-by-one error). Also add a _bad test that with --output-numerate 1 you get a too many numerated files error.

2. Add the option to V3Options.* (look for another integer example), and document in bin/verilator.

3. When V3OutFileC currently creates a file, pass a new argument saying if numeration should apply. If it does, assign a number rather than use the passed filename. Update all callers to pass the flag based on table above.

4. Check before end of main code in Verilator.cpp we didn't exceed the number of files.

5. I think the generated makefiles will end up ok as they will use the numerated file that OutFileC made, but not positive.

RE: Building Verilator with Bazel - Added by Kevin Kiningham over 1 year ago

Yes. The build systems I've used to date always would automatically use the dependencies Verilator/the simulator found. I understand you could express them to the build system, but this is duplicating information already expressed in Verilog and I hate duplication perhaps to a fault.

I see how that works for Verilator's output, but how does the build system tell Verilator what files to look at? Do you just pass all of the Verilog files in the project to Verilator? If that's the case, I could do the same in Bazel; in that case only duplication between Verilog and the BUILD file would be the names of the modules. The reason I track dependency information between Verilog files is so that I can pass a smaller set of files to Verilator.

Also, Bazel is smart enough to avoid recompiling things if the files contents don't change (it uses hashes not timestamps) and can prune irrelevant dependencies for C++, so I don't really need to rely on Verilator to tell me the dependencies between the output files. I just compile all of the Verilator output into one giant static library and let Bazel figure out the rest. The only problem is that Verilator can output different files depending on what optimization level is used, so the input to this rule has to be manually adjusted. If Verilator were able to output a consistent set of files, this limitation would go away.

However your primary point is right, we should leave the {prefix}.h file alone, and rename most the rest.

Hm, I'm still not sure. In my project I have a module "A" with a register declared using /* verilator public */ that is instantiated inside of the top module "tb". If I only include the "Vtb.h" file, I don't currently get the class definition of Vtb_A - I have to include "Vtb_A.h" for that. What would I include in this scenario?

RE: Building Verilator with Bazel - Added by Wilson Snyder over 1 year ago

Our and "typical" build system pass Verilator only the top module name and appropriate flags, and verilator figures out all other dependencies. The build tool then on the next build checks for any of those dependencies or build rules changing (using hashes), and then if needed calls Verilator, which updates dependencies for next build's calculations.

I'm still not sure. In my project I have a module "A" with a register declared using /* verilator public */ that is instantiated inside of the top module "tb". If I only include the "Vtb.h" file, I don't currently get the class definition of Vtb_A - I have to include "Vtb_A.h" for that. What would I include in this scenario?

Could you use DPI-C routines instead of public? (Faster and simulator portable.) Alternatively maybe all public modules don't rename, only nonpublic and any output-split follow-on files.

RE: Building Verilator with Bazel - Added by Kevin Kiningham over 1 year ago

Our and "typical" build system pass Verilator only the top module name and appropriate flags, and verilator figures out all other dependencies. The build tool then on the next build checks for any of those dependencies or build rules changing (using hashes), and then if needed calls Verilator, which updates dependencies for next build's calculations.

Just to make sure I understand, are these the steps you're describing?

1) Run Verilator over the whole project and let it figure out the dependency graph between modules, starting from the specified top module. Verilator produces a file that describes this dependency graph, along with generated C++ files. 2) On a rebuild, the build system first checks if the dependency graph does not exist or has changed. If so, it re-runs 1. If it hasn't, the build system checks if any of the transitive dependencies of the top module have changed. If so, it re-runs 1. If not, exit.

If that's the case, tell me if the following scenario is correct:

A project has the files

top.sv        // declares top, instantiates m
m.sv          // declares m, includes inc.sv
lib/inc.sv

and uses the flags ``--cc top.sv --top-module top -y lib``. Your build system is run, creating a dependency graph "top.sv -> m.sv -> lib/inc.sv". A developer adds a file "lib/m.sv" that contains a new module m which contains a syntax error, but changes nothing else. When your build system is re-run, the build system reads the dependency graph, sees that none of the named files have changed, and exits. When a developer cleans the build and re-runs it, verilator searches the "lib" directory before "." and exits with an error since "lib/m.sv" contains an error. If "lib/m.sv" contained a subtle logic error rather than a syntax error (perhaps an old file was reintroduced to the build, for example), this could cause a subtle error in the generated simulator. In other words, the behavior before and after a clean can be different, which breaks hermeticity.

Could you use DPI-C routines instead of public?

Unfortunately, not easily for my use case (or at least I don't see an obvious way).

Alternatively maybe all public modules don't rename, only nonpublic and any output-split follow-on files.

That sounds reasonable. If a module contains a /* verilator public */ annotation, will there always be a .h file generated?

RE: Building Verilator with Bazel - Added by Wilson Snyder over 1 year ago

The case you describe is protected by erroring (outside Verilator) if the repository ever attempts to contains two identical basefile names of sources (c/h/v/etc) in different directories (within some project scope). This is good protection anyways as once a physical tool did what was unexpected and grabbed the wrong directory, finding an old module and bad silicon would have been the result but for a final formal compare.

Actually now that I mention it, perhaps Verilator should have a similar suppressable warning? This of course doesn't protect directories Verilator doesn't know about, so doen't replace a global check.

That sounds reasonable. If a module contains a /* verilator public */ annotation, will there always be a .h file generated?

Correct. So basically the rule is any user-expected generated cpp/h file is unrenamed, all else are. Plan good to go?

RE: Building Verilator with Bazel - Added by Kevin Kiningham over 1 year ago

The case you describe is protected by erroring (outside Verilator) if the repository ever attempts to contains two identical basefile names of sources (c/h/v/etc) in different directories (within some project scope).

So you can't have two files with the same name anywhere in the project? That doesn't seem like a very scalable solution, but each to their own. What happens if you have two third party dependencies that have files with the same name?

This is good protection anyways as once a physical tool did what was unexpected and grabbed the wrong directory, finding an old module and bad silicon would have been the result but for a final formal compare.

This isn't an issue if your dependency relationships are explicitly declared.

Actually now that I mention it, perhaps Verilator should have a similar suppressable warning?

Maybe make it opt-in?

So basically the rule is any user-expected generated cpp/h file is unrenamed, all else are. Plan good to go?

Can we nail down exactly which files are considered "user-expected"? For example, if a public module A is instantiated inside of another module B, then both A and B's header file should be considered public, correct? If that's the case, then adding a single declaration can change the output filenames for a large number of files. That seems undesirable.

RE: Building Verilator with Bazel - Added by Wilson Snyder over 1 year ago

We always prefix IP modules if the vendor didn't.

Can we nail down exactly which files are considered "user-expected"? For example, if a public module A is instantiated inside of another module B, then both A and B's header file should be considered public, correct? If that's the case, then adding a single declaration can change the output filenames for a large number of files. That seems undesirable.

I don't understand your point. Changing, A, B, or any module above A or B will change the filenames, I don't see any way around that as you need to appropriately include A, B and the hierarchy to get to them.

RE: Building Verilator with Bazel - Added by Kevin Kiningham over 1 year ago

Suppose I have the file following file top.sv compiled with "--cc top.sv --top-module top":

module m0; reg a; endmodule
module m1; m0 m0_inst; endmodule
module m2; m1 m1_inst; endmodule
module top; m2 m2_inst; endmodule

So the hierarchy is "top->m2->m1->m0". The current set of (possible) output files is something like

Vtop.cpp
Vtop.h
Vtop_top.cpp
Vtop_top.h
Vtop_m2.cpp
Vtop_m2.h
Vtop_m1.cpp
Vtop_m1.h
Vtop_m0.cpp
Vtop_m0.h
// Plus extra

But the set of actually produced files can change depending on optimization decisions (for example, with the current version, Verilator only produces Vtop.cpp and Vtop.h)

As far as I understand, you're proposing the output set to be renamed to something like

Vtop.cpp
Vtop.h
Vtop__0.cpp
Vtop__0.h
Vtop__1.cpp
Vtop__1.h
Vtop__2.cpp
Vtop__2.h
Vtop__3.cpp
Vtop__3.h
// Plus extra

But if I change the code to add a /* verilator public */ directive on register a, all of the modules will be considered public. That means the output set will again be

Vtop.cpp
Vtop.h
Vtop_top.cpp
Vtop_top.h
Vtop_m2.cpp
Vtop_m2.h
Vtop_m1.cpp
Vtop_m1.h
Vtop_m0.cpp
Vtop_m0.h
// Plus extra

In other words, a single /* verilator public */ has made a big change in the set of output files. Since my aim is to make the set of output files more stable, that seems undesirable.

A different approach would be to just always create the original set, even if some of the files were empty. That would solve my problem.

Does that make sense?

RE: Building Verilator with Bazel - Added by Wilson Snyder over 1 year ago

The files created would be

Vtop.cpp
Vtop.h
Vtop_0.cpp
Vtop_top.h
Vtop_1.cpp
Vtop_m2.h
Vtop_2.cpp
Vtop_m1.h
Vtop_3.cpp
Vtop_m0.h

Because the "top" files are always made, and the .h's because the publics and all other files are internal only. If a public file or hierarchy changes you must edit your rules by definition of needing those headers, but no others. If nothing is made public no design change will change the output files.

RE: Building Verilator with Bazel - Added by Wilson Snyder over 1 year ago

Actually double underscores, Vtop__0.cpp, to prevent conflicting with module named top_0.

RE: Building Verilator with Bazel - Added by Kevin Kiningham over 1 year ago

If a public file or hierarchy changes you must edit your rules by definition of needing those headers, but no others.

I don't think that's always true. The /* verilator public */ annotation could be extraneous; i.e. I could add the annotation but not actually use it in my testbench. That might seem silly, but if multiple testbenches depend on the same verilog file, it would be nice not to have to update the build rules for all of them. For example, I might add a public annotation so a unit test can read some internal state, but a larger integration test shouldn't have to care.

Also, the build rules would only have to be edited under your proposal; under my alternative even if the testbench uses the header, I'd only need to edit the testbench to add the include, but the build rules would stay the same.

Let me try to be more explicit about why I'm pushing for a different approach. Right now, under a slightly modified version of the the senerio I described in my previous post, the build rules would look something like the following:

verilog_library(
    name = "m",
    srcs = ["m.sv"],
    modules = ["m0", "m1", "m2"],
)

verilog_library(
    name = "top" 
    srcs = ["top.sv"],
    modules = ["top"],
    deps = [":m"],
)

cc_test(
    name = "testbench",
    srcs = ["top_tb.cpp"],
    deps = [":top"],
)

"verilog_library" is a custom rule that says "any rule that depends on this rule can read the set of header files produced by running verilator over my source files and my dependency's source files". In order to implement this rule, I need to be able to write a mapping from the information passed to the rule to the set of output files actually produced by Verilator. Under your proposal that's basically impossible; the rule needs to also know which modules are considered public. I could add that information, but that would mean anyone depending on that rule would have to manually keep their list of public modules in sync with their dependencies (since any module that instantiates a public module would then also be public) even if they don't actually use the header files produced by the module in their testbench. That sort of "infectious" behaviour seems pretty undesirable, and I think the end result would be people just declaring all modules to be public in order to reduce the churn in the build rules.

On the other hand, under the other approach I mentioned writing this mapping is super straightforward; it's just the list we mentioned before ("{prefix}.cpp, "{prefix}{module}.cpp", etc).

Let me know if my reasoning doesn't seem clear - I'll try to come up with a more detailed example.

To reverse the question a bit, is there an advantage to not doing it the way I'm proposing?

RE: Building Verilator with Bazel - Added by Wilson Snyder over 1 year ago

What you suggest will break once parameters are used.

RE: Building Verilator with Bazel - Added by Kevin Kiningham over 1 year ago

Assuming you're talking about Verilog parameters changing the hierarchy, would that not be true for your proposal as well? For example, say I have either "A -> B1 -> C" or "A -> B2 -> C" depending on a parameter, and C has a /* verilator public */ annotation. Under your proposal either "VA_B1.h" or "VA_B2.h" would be generated, depending on the parameter.

Regardless, if you have a parameter that controls the hierarchy it seems very likely it would also affect the dependency graph (this is certainly true if each module is in a separate file.) If you prefer dependencies to be implicit (as it sounds like you do), automatically generating the dependency graph makes this easy to deal with. If you prefer dependencies to be explicit (as is required in Bazel) you're left either updating the rules manually or encoding some of the instantiation logic in the build system. I've opted for the latter, and Bazel has enough tools around project configuration that I haven't found it to be an issue in practice. Additionally, If you get the configuration logic wrong it results in a build failure (since Verilator would be looking for a file that's not visible) so it's not a very high risk problem either.

Is there another way this breaks that I'm missing?

RE: Building Verilator with Bazel - Added by Wilson Snyder over 1 year ago

Module a with two different parameters passed in needs two differently named h and cpp files.

RE: Building Verilator with Bazel - Added by Kevin Kiningham over 1 year ago

Your solution breaks in an identical way if /* verilator public */ is used.

RE: Building Verilator with Bazel - Added by Wilson Snyder over 1 year ago

Correct, didn't intend to imply otherwise. My point was a counter to your proposal of determining output names.

Perhaps the course is for you prototype something that works for you internally, and after a few months of your getting the kinks out suggest the appropriate patches.

RE: Building Verilator with Bazel - Added by Kevin Kiningham 5 months ago

Quick update for anyone who's come across this thread: I've improved the Bazel rules such that they no longer require a fork of Verilator. If you find them useful, feel free to give feedback or open an issue on it's Github page !

    (1-23/23)