Regression Tests¶
Regression tests are tests that are repeatedly executed to assess functionality. Each regression test targets a specific function of Odin II. There are two main components of a regression test; benchmarks and a configuration file. The benchmarks are comprised of verilog files, input vector files and output vector files. The configuration file calls upon each benchmark and synthesizes them with different architectures. The current regression tests of Odin II can be found in regression_test/benchmark.
Benchmarks¶
Benchmarks are used to test the functionality of Odin II and ensure that it runs properly.
Benchmarks of Odin II can be found in regression_test/benchmark/verilog/any_folder.
Each benchmark is comprised of a verilog file, an input vector file, and an output vector file.
They are called upon during regression tests and synthesized with different architectures to be compared against the expected results.
These tests are useful for developers to test the functionality of Odin II after implementing changes.
The command make test runs through all these tests, comparing the results to previously generated results, and should be run through when first installing.
Unit Benchmarks¶
Unit benchmarks are the simplest of benchmarks. They are meant to isolate different functions of Odin II. The goal is that if it does not function properly, the error can be traced back to the function being tested. This cannot always be achieved as different functions depend on others to work properly. It is ideal that these benchmarks test bit size capacity, erroneous cases, as well as standards set by the IEEE Standard for Verilog® Hardware Description Language - 2005.
Micro Benchmarks¶
Micro benchmarks are precise, like unit benchmarks, however are more syntactic. They are meant to isolate the behaviour of different functions. They trace the behaviour of functions to ensure they adhere to the IEEE Standard for Verilog® Hardware Description Language - 2005. Like unit benchmarks, they should check erroneous cases and behavioural standards set by the IEEE Standard for Verilog® Hardware Description Language - 2005.
Macro Benchmarks¶
Macro benchmarks are more realistic tests that incorporate multiple functions of Odin II. They are intended to simulate real-user behaviour to ensure that functions work together properly. These tests are designed to test things like syntax and more complicated standards set by the IEEE Standard for Verilog® Hardware Description Language - 2005.
External Benchmarks¶
External benchmarks are benchmarks created by outside users to the project. It is possible to pull an outside directory and build them on the fly thus creating a benchmark for Odin II.
Creating Regression Tests¶
New Regression Test Checklist¶
Create benchmarks here
Create configuration file here
Create a folder in the task directory for the configuration file here
Generate the results here
Add the task to a suite (large suite if generating the results takes longer than 3 minutes, otherwise put in light suite) here
Update the documentation by providing a summary in Regression Test Summary section and updating the Directory Tree here
New Benchmarks added to Regression Test Checklist¶
Include¶
verilog file
input vector file
expected output vector file
configuration file (conditional)
architecture file (optional)
Creating Benchmarks¶
If only a few benchmarks are needed for a PR, simply add the benchmarks to the appropriate set of regression tests. The Regression Test Summary summarizes the target of each regression test which may be helpful.
The standard of naming the benchmarks are as follows:
verilog file: meaningful_title.v
input vector file: meaningful_title_input
output vector file: meaningful_title_output
If the tests needed do not fit in an already existing set of regression tests or need certain architecture(s), create a separate folder in the verilog directory and label appropriately. Store the benchmarks in that folder. Add the architecture (if it isn’t one that already exists) to ../vtr_flow/arch.
NOTE
If a benchmark fails and should pass, include a $display statement in the verilog file in the following format:
$display("Expect::FUNCTION < message >);The function should be in all caps and state what is causing the issue. For instance, if else if was behaving incorrectly it should read ELSE_IF. The message should illustrate what should happen and perhaps a suggestion in where things are going wrong.
Creating a Configuration File¶
A configuration file is only necessary if the benchmarks added are placed in a new folder. The configuration file is where architectures and commands are specified for the synthesis of the benchmarks. The configuration file must be named task.conf. The following is an example of a standard task.conf (configuration) file:
########################
# <title> benchmarks config
########################
# commands
regression_params=--include_default_arch
script_synthesis_params=--time_limit 3600s
script_simulation_params=--time_limit 3600s
simulation_params= -L reset rst -H we
# setup the architecture (standard architectures already available)
archs_dir=../vtr_flow/arch/timing
arch_list_add=k6_N10_40nm.xml
arch_list_add=k6_N10_mem32K_40nm.xml
arch_list_add=k6_frac_N10_frac_chain_mem32K_40nm.xml
# setup the circuits
circuits_dir=regression_test/benchmark/verilog/
circuit_list_add=<verilog file group>/*.vh
circuit_list_add=<verilog file group>/*.v
synthesis_parse_file=regression_test/parse_result/conf/synth.toml
simulation_parse_file=regression_test/parse_result/conf/sim.toml
The following key = value are available for configuration files:
| key | following argument |
|---|---|
| circuits_dir | < path/to/circuit/dir > |
| circuit_list_add | < circuit file path relative to [circuits_dir] > |
| archs_dir | < path/to/arch/dir > |
| arch_list_add | < architecture file path relative to [archs_dir] > |
| synthesis_parse_file | < path/to/parse/file > |
| simulation_parse_file | < path/to/parse/file > |
| script_synthesis_params | [see exec_wrapper.sh options] |
| script_simulation_params | [see exec_wrapper.sh options] |
| synthesis_params | [see Odin options] |
| simulation_params | [see Odin options] |
| regression_params | [see Regression Parameters bellow] |
Regression Parameters:
--verbosedisplay error logs after batch of tests--concat_circuit_listconcatenate the circuit list and pass it straight through to odin--generate_benchgenerate input and output vectors from scratch--disable_simulationdisable the simulation for this task--disable_parallel_jobsdisable running circuit/task pairs in parallel--randomizeperform a dry run randomly to check the validity of the task and flow |--regenerate_expectationregenerate expectation and override the expected value only if there’s a mismatch |--generate_expectationgenerate the expectation and override the expectation file |
Creating a Task¶
The following diagram illustrates the structure of regression tests. Each regression test needs a corresponding folder in the task directory containing the configuration file. The <task display name> should have the same name as the verilog file group in the verilog directory. This folder is where the synthesis results and simulation results will be stored. The task diplay name and the verilog file group should share the same title.
└── odin_ii
└── regression_test
└── benchmark
├── task
│ └── < task display name >
│ ├── [ simulation_result.json ]
│ ├── [ synthesis_result.json ]
│ └── task.conf
└── verilog
└── < verilog file group >
├── *.v
└── *.vh
Creating a Complicated Task¶
There are times where multiple configuration files are needed in a regression test due to different commands wanted or architectures. The task cmd_line_args is an example of this. If that is the case, each configuration file will still need its own folder, however these folders should be placed in a parent folder.
└── odin_ii
└── regression_test
└── benchmark
├── task
│ └── < parent task display name >
| ├── < task display name >
│ │ ├── [ simulation_result.json ]
│ │ ├── [ synthesis_result.json ]
| │ └── task.conf
│ └── < task display name >
│ . ├── [ simulation_result.json ]
│ . ├── [ synthesis_result.json ]
| . └── task.conf
└── verilog
└── < verilog file group >
├── *.v
└── *.vh
Creating a Suite¶
Suites are used to call multiple tasks at once. This is handy for regenerating results for multiple tasks. In the diagram below you can see the structure of the suite. The suite contains a configuration file that calls upon the different tasks named task_list.conf.
└── odin_ii
└── regression_test
└── benchmark
├── suite
│ └── < suite display name >
| └── task_list.conf
├── task
│ ├── < parent task display name >
| │ ├── < task display name >
│ │ │ ├── [ simulation_result.json ]
│ │ │ ├── [ synthesis_result.json ]
| │ │ └── task.conf
│ │ └── < task display name >
│ │ ├── [ simulation_result.json ]
│ │ ├── [ synthesis_result.json ]
| │ └── task.conf
│ └── < task display name >
│ ├── [ simulation_result.json ]
│ ├── [ synthesis_result.json ]
| └── task.conf
└── verilog
└── < verilog file group >
├── *.v
└── *.vh
In the configuration file all that is required is to list the tasks to be included in the suite with the path. For example, if the wanted suite was to call the binary task and the operators task, the configuration file would be as follows:
regression_test/benchmark/task/operators
regression_test/benchmark/task/binary
For more examples of task_list.conf configuration files look at the already existing configuration files in the suites.
Regenerating Results¶
WARNING
BEFORE regenerating the result, run
make testto ensure any changes in the code don’t affect the results of benchmarks beside your own. If they do, the failing benchmarks will be listed.
Regenerating results is necessary if any regression test is changed (added benchmarks), if a regression test is added, or if a bug fix was implemented that changes the results of a regression test. For all cases, it is necessary to regenerate the results of the task corresponding to said change. The following commands illustrate how to do so:
make sanitize
then: where N is the number of processors in the computer, and the path following -t ends with the same name as the folder you placed
./verify_odin.sh -j N --regenerate_expectation -t regression_test/benchmark/task/<task_display_name>
NOTE
DO NOT run the
make sanitizeif regenerating the large test. It is probable that the computer will not have enough ram to do so and it will take a long time. Instead runmake build
For more on regenerating results, refer to the Verify Script section.
Regression Test Summaries¶
c_functions¶
This regression test targets c functions supported by Verilog such as clog_2.
cmd_line_args¶
This is a more complicated regression test that incorporates multiple child tasks. It targets different commands available in Odin II. Although it doesn’t have a dedicated set of benchmarks in the verilog folder, the configuration files call on different preexisting benchmarks.
FIR¶
FIR is an acronym for “Finite Impulse Response”. These benchmarks were sourced from Layout Aware Optimization of High Speed Fixed Coefficient FIR Filters for FPGAs. They test a method of implementing high speed FIR filters on FPGAs discussed in the paper.
full¶
The full regression test is designed to test real user behaviour.
It does this by simulating flip flop, muxes and other common uses of Verilog.
large¶
This regression test targets cases that require a lot of ram and time.
micro¶
The micro regression test targets hards blocks and pieces that can be easily instantiated in architectures.
mixing_optimization¶
The mixing optimization regression test targets mixing implementations for operations implementable in hard blocks and their soft logic counterparts that can be can be easily instantiated in architectures. The tests support extensive command line coverage, as well as provide infrastructure to enable the optimization from an .xml configuration file, require for using the optimization as a part of VTR synthesis flow.
operators¶
This regression test targets the functionality of different operators. It checks bit size capacity and behaviour.
syntax¶
The syntax regression test targets syntactic behaviour. It checks that functions work cohesively together and adhere to the verilog standard.
keywords¶
This regression test targets the function of keywords. It has a folder or child for each keyword containing their respective benchmarks. Some folders have benchmarks for two keywords like task_endtask because they both are required together to function properly.
preprocessor¶
This set of regression test includes benchmarks targetting compiler directives available in Verilog.
Regression Tests Directory Tree¶
benchmark
├── suite
│ ├── complex_synthesis_suite
│ │ └── task_list.conf
│ ├── full_suite
│ │ └── task_list.conf
│ ├── heavy_suite
│ │ └── task_list.conf
│ └── light_suite
│ └── task_list.conf
├── task
│ ├── arch_sweep
│ │ ├── synthesis_result.json
│ │ └── task.conf
│ ├── c_functions
│ │ └── clog2
│ │ ├── simulation_result.json
│ │ ├── synthesis_result.json
│ │ └── task.conf
│ ├── cmd_line_args
│ │ ├── batch_simulation
│ │ │ ├── simulation_result.json
│ │ │ ├── synthesis_result.json
│ │ │ └── task.conf
│ │ ├── best_coverage
│ │ │ ├── simulation_result.json
│ │ │ ├── synthesis_result.json
│ │ │ └── task.conf
│ │ ├── coverage
│ │ │ ├── simulation_result.json
│ │ │ ├── synthesis_result.json
│ │ │ └── task.conf
│ │ ├── graphviz_ast
│ │ │ ├── synthesis_result.json
│ │ │ └── task.conf
│ │ ├── graphviz_netlist
│ │ │ ├── synthesis_result.json
│ │ │ └── task.conf
│ │ └── parallel_simulation
│ │ ├── simulation_result.json
│ │ ├── synthesis_result.json
│ │ └── task.conf
│ ├── FIR
│ │ ├── simulation_result.json
│ │ ├── synthesis_result.json
│ │ └── task.conf
│ ├── fpu
│ │ └── hardlogic
│ │ ├── simulation_result.json
│ │ ├── synthesis_result.json
│ │ └── task.conf
│ ├── full
│ │ ├── simulation_result.json
│ │ ├── synthesis_result.json
│ │ └── task.conf
│ ├── keywords
│ │ ├── always
│ │ ├── and
│ │ ├── assign
│ │ ├── at_parenthathese
│ │ ├── automatic
│ │ ├── begin_end
│ │ ├── buf
│ │ ├── case_endcase
│ │ ├── default
│ │ ├── defparam
│ │ ├── else
│ │ ├── for
│ │ ├── function_endfunction
│ │ ├── generate
│ │ ├── genvar
│ │ ├── if
│ │ ├── initial
│ │ ├── inout
│ │ ├── input_output
│ │ ├── integer
│ │ ├── localparam
│ │ ├── macromodule
│ │ ├── nand
│ │ ├── negedge
│ │ ├── nor
│ │ ├── not
│ │ ├── or
│ │ ├── parameter
│ │ ├── posedge
│ │ ├── reg
│ │ ├── signed_unsigned
│ │ ├── specify_endspecify
│ │ ├── specparam
│ │ ├── star
│ │ ├── task_endtask
│ │ ├── while
│ │ ├── wire
│ │ ├── xnor
│ │ └── xor
│ ├── koios
│ │ ├── synthesis_result.json
│ │ └── task.conf
│ ├── large
│ │ ├── synthesis_result.json
│ │ └── task.conf
│ ├── micro
│ │ ├── simulation_result.json
│ │ ├── synthesis_result.json
│ │ └── task.conf
│ ├── mixing_optimization
│ │ ├── mults_auto_full
│ │ │ ├── simulation_result.json
│ │ │ │── synthesis_result.json
│ │ │ └── task.conf
│ │ ├── mults_auto_half
│ │ │ ├── simulation_result.json
│ │ │ │── synthesis_result.json
│ │ │ └── task.conf
│ │ ├── mults_auto_none
│ │ │ ├── simulation_result.json
│ │ │ │── synthesis_result.json
│ │ │ └── task.conf
│ │ ├── config_file_half
│ │ │ ├── config_file_half.xml
│ │ │ ├── simulation_result.json
│ │ │ │── synthesis_result.json
│ │ │ └── task.conf
│ ├── operators
│ │ ├── simulation_result.json
│ │ ├── synthesis_result.json
│ │ └── task.conf
│ ├── preprocessor
│ │ ├── simulation_result.json
│ │ ├── synthesis_result.json
│ │ └── task.conf
│ ├── syntax
│ │ ├── simulation_result.json
│ │ ├── synthesis_result.json
│ │ └── task.conf
│ └── vtr
│ ├── synthesis_result.json
│ └── task.conf
│
├── third_party
│ └── SymbiFlow
│ ├── build.sh
│ └── task.mk
└── verilog
├── FIR
├── c_functions
├── common
├── full
├── keywords
├── large
├── micro
├── operators
├── preprocessor
└── syntax