Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V3Hashed.cpp Called isIdentical on non-hashed nodes, from Gate dedupe() #1475

Closed
veripoolbot opened this issue Jun 27, 2019 · 20 comments
Closed
Assignees
Labels
area: wrong runtime result Issue involves an incorrect runtine result from Verilated model resolution: fixed Closed; fixed

Comments

@veripoolbot
Copy link
Contributor


Author Name: Øyvind Harboe
Original Redmine Issue: 1475 from https://www.veripool.org

Original Assignee: Wilson Snyder (@wsnyder)


I'm getting the error below in BOLD when I invoke Verilator. DressRehearsalTestBench.v is generated by Chisel: https://chisel.eecs.berkeley.edu/

I'm not sure how to proceed with this bug-report as I haven't been able to reduce DressRehearsalTestBench.v to the point where I can include it in the bug-report.

$ verilator --cc DressRehearsalTestBench.v --assert -Wno-fatal -Wno-WIDTH -Wno-STMTDLY -O1 --top-module DressRehearsalTestBench +define+TOP_TYPE=VDressRehearsalTestBench +define+PRINTF_COND=\!DressRehearsalTestBench.reset +define+STOP_COND=\!DressRehearsalTestBench.reset -CFLAGS "-Wno-undefined-bool-conversion -O1 -DTOP_TYPE=VDressRehearsalTestBench -DVL_USER_FINISH -include VDressRehearsalTestBench.h" -Mdir /[deleted]/test_run_dir/dressrehearsal -f /[deleted]/dressrehearsal/black_box_verilog_files.f --exe /[deleted]/test_run_dir/dressrehearsal/DressRehearsalTestBench-harness.cpp --trace
%Error: Internal Error: DressRehearsalTestBench.v:18956: *../V3Hashed.cpp:134: Called isIdentical on non-hashed nodes*
%Error: Internal Error: See the manual and http://www.veripool.org/verilator for more assistance.
%Error: Command Failed /usr/local/bin/verilator_bin --cc DressRehearsalTestBench.v --assert -Wno-fatal -Wno-WIDTH -Wno-STMTDLY -O1 --top-module DressRehearsalTestBench \+define\+TOP_TYPE\=VDressRehearsalTestBench \+define\+PRINTF_COND\=\!DressRehearsalTestBench.reset \+define\+STOP_COND\=\!DressRehearsalTestBench.reset -CFLAGS -Wno-undefined-bool-conversion\ -O1\ -DTOP_TYPE\=VDressRehearsalTestBench\ -DVL_USER_FINISH\ -include\ VDressRehearsalTestBench.h -Mdir /[deleted]/test_run_dir/dressrehearsal -f /[deleted]//test_run_dir/dressrehearsal/black_box_verilog_files.f --exe /[deleted]/test_run_dir/dressrehearsal/DressRehearsalTestBench-harness.cpp --trace

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Øyvind Harboe
Original Date: 2019-06-27T18:21:11Z


This is with 4.016 as well as 4.014:

$ verilator --version
Verilator 4.014 2019-05-08 rev UNKNOWN_REV

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Wilson Snyder (@wsnyder)
Original Date: 2019-06-27T22:04:15Z


There was recently a similar error resulting from parameters, are there parameters in DressRehersalTestBench.v?

What is near DressRehersalTestBench.v line 18956?

If reduction of code makes the bug go away, try the -Oi flag to disable inlining. (It might be the smaller code then gets inlined, hiding the bug.)

Try using --debug. Let us know maybe the last 10 lines of the output.

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Øyvind Harboe
Original Date: 2019-06-27T22:13:40Z


     if (_T_40457) begin
       routeOut_1 <= io_leArray_routeOut_1;
     end -- line 18956

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Øyvind Harboe
Original Date: 2019-06-27T22:17:00Z


The parameterization happens in Chisel, so Chisel spits out unparameterized files, so the only parameterization I have is in a blackbox inferred RAM Verilog file. It's nowhere near the failure location.

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Øyvind Harboe
Original Date: 2019-06-27T22:21:37Z


This is the only other file in the design in addition to DressRehearsalTestBench.v

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Øyvind Harboe
Original Date: 2019-06-27T22:42:51Z


I was able to run a bisection to find the first failing version of Verilator:

$ git bisect bad
597d28b505ec16f571685347a0050b279ad5781f is the first bad commit
commit 597d28b505ec16f571685347a0050b279ad5781f
Author: Wilson Snyder <wsnyder@wsnyder.org>
Date:   Thu Feb 1 21:32:58 2018 -0500

     Fix internals to make null-pointer-check clean. Also add more const's. No functional change intended, but likely something will break.

:100644 100644 13dc1c06c4f317112f803c1d862fb3be413cf9fc cf9ec6f382c4290fc0014026d2d60edd44754dee M      Changes
:100644 100644 bdab2c727715268b14c46d18df3d11d125868b20 e6e079c4417b9fd6c7c195a29761ee9440faecfe M      configure.ac
:040000 040000 15ec2fa9ddf78dd976b305ce718e9efac2a97d83 4af235d50eb572bd88ce50a2e04ecafcf13bca4b M      src

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Wilson Snyder (@wsnyder)
Original Date: 2019-06-27T22:54:24Z


Bisecting was a good idea, unfortunately that's was a hyper large change, though I wouldn't have expected those edits to result in this error.

What does "verilator --debug --gdbbt" give as a backtrace?

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Øyvind Harboe
Original Date: 2019-06-27T23:28:58Z


Is it possible to split that commit into a sequence of commits, purely for the purpose of bisecting it further or is it necessarily a single big commit?

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Øyvind Harboe
Original Date: 2019-06-27T23:34:31Z


Here's the last lines when running with "--debug --gdbgt":

- V3Ast.cpp:1050:     Dumping /home/[deleted]/test_run_dir/dressrehearsal/VDressRehearsalTestBench_083_subst.tree
- V3Const.cpp:2625:   constifyCpp: 
- V3Ast.cpp:1050:     Dumping /home/[deleted]/test_run_dir/dressrehearsal/VDressRehearsalTestBench_084_const_cpp.tree
- V3Dead.cpp:451:     deadifyAll: 
- V3Ast.cpp:1050:     Dumping /home/[deleted]/test_run_dir/dressrehearsal/VDressRehearsalTestBench_085_deadAll.tree
- V3Reloop.cpp:260:   reloopAll: 
- V3Depth.cpp:177:    depthAll: 
- V3Branch.cpp:138:   branchAll: 
- V3Cast.cpp:184:     castAll: 
- V3Ast.cpp:1050:     Dumping /home/[deleted]/test_run_dir/dressrehearsal/VDressRehearsalTestBench_088_cast.tree
- V3CCtors.cpp:148:   cctorsAll: 
- V3EmitCInlines.cpp:94:emitcInlines: 
- V3EmitCSyms.cpp:664:emitcSyms: 
- V3EmitC.cpp:3207:   emitcTrace: 
- V3EmitC.cpp:3193:   emitc: 
- V3EmitXml.cpp:334:  emitxml: 
- V3StatsReport.cpp:241:statsReport: 
- V3EmitMk.cpp:247:   emitmk: 
- V3Os.cpp:57:        export VERILATOR_ROOT=/usr/local/share/verilator # Hardcoded at build time
- V3Ast.cpp:1050:     Dumping /home/[deleted]/test_run_dir/dressrehearsal/VDressRehearsalTestBench_990_final.tree
- Verilator.cpp:664:  Done, Exiting...
[Inferior 1 (process 9814) exited normally]
No stack.

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Wilson Snyder (@wsnyder)
Original Date: 2019-06-28T00:06:53Z


The logfile you attached showed no errors. Perhaps you're still in the passing bisect?

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Øyvind Harboe
Original Date: 2019-06-28T07:36:23Z


I'm afraid that I'm only seeing the crash without "--debug --gdbgt", I checked.

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Øyvind Harboe
Original Date: 2019-06-28T07:36:56Z


I tried with valgrind, the crash was reproducible, but it didn't report anything.

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Wilson Snyder (@wsnyder)
Original Date: 2019-06-28T09:59:53Z


Presumably it's --debug that is hiding the issue as the code compiles differently.

Try --debugi 3. Also edit line 135 of V3Hashed.cpp with attached patch.

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Øyvind Harboe
Original Date: 2019-06-28T10:33:36Z


Output w/patch:

FIXME1: VARREF 0x56443a087580 <e2076892#> {e18755} u4=0x1651682 @dt=0x5644384fc430@(G/w1)  __PVT__leArrayScanChain_io_leArray_enable [RV] <- VARSCOPE 0x56443d41a5a0 <e1041076> {e27500} u1=0x56443aa6a9d0 u2=0x1 @dt=0x5644384fc430@(G/w1)  TOP.DressRehearsalTestBench.aptos->__PVT__leArrayScanChain_io_leArray_enable -> VAR 0x564439ea87e0 <e621691> {e27500} @dt=0x5644384fc430@(G/w1)  __PVT__leArrayScanChain_io_leArray_enable WIRE
FIXME2: VARREF 0x564439c1b350 <e2077116#> {e18749} @dt=0x5644384fc430@(G/w1)  __PVT__leArrayScanChain_io_leArray_enable [RV] <- VARSCOPE 0x56443d41a5a0 <e1041076> {e27500} u1=0x56443aa6a9d0 u2=0x1 @dt=0x5644384fc430@(G/w1)  TOP.DressRehearsalTestBench.aptos->__PVT__leArrayScanChain_io_leArray_enable -> VAR 0x564439ea87e0 <e621691> {e27500} @dt=0x5644384fc430@(G/w1)  __PVT__leArrayScanChain_io_leArray_enable WIRE
%Error: Internal Error: fail.v:18749: ../V3Hashed.cpp:143: Called isIdentical on non-hashed nodes
                         ... See the manual and http://www.veripool.org/verilator for more assistance.


Output with "--debugi 3":

- V3Active.cpp:440:   activeAll: 
- V3Ast.cpp:1050:     Dumping /home/oyvind/ascenium/aptos/test_run_dir/dressrehearsal/VDressRehearsalTestBench_045_active.tree
- V3Split.cpp:1015:   splitAlwaysAll: 
- V3Ast.cpp:1050:     Dumping /home/oyvind/ascenium/aptos/test_run_dir/dressrehearsal/VDressRehearsalTestBench_046_split.tree
- V3SplitAs.cpp:219:  splitAsAll: 
- V3Ast.cpp:1050:     Dumping /home/oyvind/ascenium/aptos/test_run_dir/dressrehearsal/VDressRehearsalTestBench_047_splitas.tree
- V3TraceDecl.cpp:333:traceDeclAll: 
- V3Ast.cpp:1050:     Dumping /home/oyvind/ascenium/aptos/test_run_dir/dressrehearsal/VDressRehearsalTestBench_048_tracedecl.tree
- V3Gate.cpp:1540:    gateAll: 
dot -Tpdf -o ~/a.pdf /home/oyvind/ascenium/aptos/test_run_dir/dressrehearsal/VDressRehearsalTestBench_049_gate_simp.dot
FIXME1: VARREF 0x560c8d9408b0 <e2076892#> {e18755} u4=0x1aa5e3a @dt=0x560c8c948c20@(G/w1)  __PVT__leArrayScanChain_io_leArray_enable [RV] <- VARSCOPE 0x560c9195dcd0 <e1041076> {e27500} u1=0x560c8f10b780 u2=0x1 @dt=0x560c8c948c20@(G/w1)  TOP.DressRehearsalTestBench.aptos->__PVT__leArrayScanChain_io_leArray_enable -> VAR 0x560c8e2f5480 <e621691> {e27500} @dt=0x560c8c948c20@(G/w1)  __PVT__leArrayScanChain_io_leArray_enable WIRE
FIXME2: VARREF 0x560c8d8ed170 <e2077116#> {e18749} @dt=0x560c8c948c20@(G/w1)  __PVT__leArrayScanChain_io_leArray_enable [RV] <- VARSCOPE 0x560c9195dcd0 <e1041076> {e27500} u1=0x560c8f10b780 u2=0x1 @dt=0x560c8c948c20@(G/w1)  TOP.DressRehearsalTestBench.aptos->__PVT__leArrayScanChain_io_leArray_enable -> VAR 0x560c8e2f5480 <e621691> {e27500} @dt=0x560c8c948c20@(G/w1)  __PVT__leArrayScanChain_io_leArray_enable WIRE
%Error: Internal Error: fail.v:18749: ../V3Hashed.cpp:143: Called isIdentical on non-hashed nodes
-node: VARREF 0x560c8d8ed170 <e2077116#> {e18749} @dt=0x560c8c948c20@(G/w1)  __PVT__leArrayScanChain_io_leArray_enable [RV] <- VARSCOPE 0x560c9195dcd0 <e1041076> {e27500} u1=0x560c8f10b780 u2=0x1 @dt=0x560c8c948c20@(G/w1)  TOP.DressRehearsalTestBench.aptos->__PVT__leArrayScanChain_io_leArray_enable -> VAR 0x560c8e2f5480 <e621691> {e27500} @dt=0x560c8c948c20@(G/w1)  __PVT__leArrayScanChain_io_leArray_enable WIRE
                         ... See the manual and http://www.veripool.org/verilator for more assistance.
- V3Ast.cpp:1050:     Dumping /home/oyvind/ascenium/aptos/test_run_dir/dressrehearsal/VDressRehearsalTestBench_990_final.tree
- V3StatsReport.cpp:241:statsReport: 
%Error: Internal Error: Aborting since under --debug
%Error: Verilator aborted.  Consider trying --debug --gdbbt
%Error: Command Failed /usr/local/bin/verilator_bin --debugi 3 --cc fail.v --assert -Wno-fatal -Wno-WIDTH -Wno-STMTDLY -O1 --top-module DressRehearsalTestBench \+define\+TOP_TYPE\=VDressRehearsalTestBench \+define\+PRINTF_COND\=\!DressRehearsalTestBench.reset \+define\+STOP_COND\=\!DressRehearsalTestBench.reset -CFLAGS -Wno-undefined-bool-conversion\ -O1\ -DTOP_TYPE\=VDressRehearsalTestBench\ -DVL_USER_FINISH\ -include\ VDressRehearsalTestBench.h -Mdir /home/oyvind/ascenium/aptos/test_run_dir/dressrehearsal -f /home/oyvind/ascenium/aptos/test_run_dir/dressrehearsal/black_box_verilog_files.f --exe /home/oyvind/ascenium/aptos/test_run_dir/dressrehearsal/DressRehearsalTestBench-harness.cpp --trace


@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Wilson Snyder (@wsnyder)
Original Date: 2019-06-28T16:45:18Z


That helps, but not enough to suggest what to fix so still think we need to get to a testcase. Basically, Verilator suspects that leArrayScanChain_io_leArray_enable feeds to some logic that is probably duplicated and can be simplified, and is getting this error trying to prove it.

Please look to see if you can make a test case that involves the signal leArrayScanChain_io_leArray_enable (probably feeding to _T_40457). Another method if you haven't tried it is to see it fail, rip out some code and if still fails keep ripping. Backup if it starts passing.

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Øyvind Harboe
Original Date: 2019-06-28T17:10:17Z


I've tried to reduce it along the lines you describe and I've failed. I'm skeptical of this apparoch because simply specifying debug output is enough to make the problem go away, which I assume, means that there's some sort of dangling reference or uninitialize data. Also, I've had a great many variations of this dress rehearsal test and it's only the exact version that I have now that gives Verilator constipation.

The failing Verilog is 2.2mBytes uncompressed and 37000 lines of code. It's a scaled down version of what I'm building, which is going to be ~500k lines of Verilog.

I can investigate the possibility of emailing you the Verilog file privately and then you can use it to debug the problem, if possible extract a test-case once you understand the problem to put into a test-suite and then delete the Verilog afterwards.

Would that be a meaningful way forward?

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Wilson Snyder (@wsnyder)
Original Date: 2019-07-22T12:47:42Z


Got a test case which shows a replacement is going into a flop which itself is then subject to replacement. This is inside the "Gate dedupe() outputs" stage of optimization.

I have ugly code which can detect this case and disable further optimizations on the substituted block, but would prefer to commit something that can continue optimization, unfortunately this requires rework of the data structures.

As a temp workaround please try commenting the block of code after printing "Gate dedupe() outputs" in V3Gate.cpp.

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Wilson Snyder (@wsnyder)
Original Date: 2019-08-04T02:01:47Z


Fixed in git towards 4.017. Thanks for the test case & patience.

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Øyvind Harboe
Original Date: 2019-08-04T10:10:02Z


Thanks for fixing this. This must have been an obscure edge case, I only ran into it that once. Given the Verilator test-suite size, I would expect new tests to be more and more obscure. I've been running with quite an old version of Verilator for a year or so, and when I was finally prompted to upgrade, this was the only snag.

I'm glad it's in the automated test-suite now.

@veripoolbot
Copy link
Contributor Author


Original Redmine Comment
Author Name: Wilson Snyder (@wsnyder)
Original Date: 2019-08-29T23:14:45Z


In 4.018.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: wrong runtime result Issue involves an incorrect runtine result from Verilated model resolution: fixed Closed; fixed
Projects
None yet
Development

No branches or pull requests

2 participants