A recent paper entitled “Finding and Understanding Bugs in FPGA Synthesis Tools” by Yann Herklotz and John Wickerson describes a new tool called Verismith that can automatically test FPGA synthesis tools. It does this by using randomly generated Verilog, and checking that the synthesised netlist is always equivalent to the original design.
Any FPGA designer relies on their synthesis tool to do the right thing. This paper argues that
as FPGAs are increasingly used as accelerators in the cloud and designs become more complex, the correctness of the synthesis tools grows in importance.
They were able to use Verismith to help them answer some interesting questions; including how many bugs could they find in various synthesis tools, and how those bugs changed over different releases of the same tools.
📷
The paper describes how they generate random Verilog in a way that doesn’t result in warnings or errors in the synthesis tools. These small programs are then synthesised by the synthesis tools under test.
They tested multiple versions of Yosys, Vivado, XST and Quartus Prime. The equivalence check was performed using Yosys and the ABC back end.
Finally, failing designs were ‘reduced’ to be able to quickly locate exactly where the problem occurred.
Verismith was run for 18000 CPU hours and discovered a number of bugs in the tools. You’ll have to check the paper for a full report, but we’re happy to say Yosys faired well, and that most of the Yosys bugs got fixed within a day of them being reported.
Their paper can be downloaded here.