GenBen (Generative Hardware Benchmark)is a new benchmark for evaluating large language models (LLMs) in hardware design. We have provided some of the problems in data.zip, along with the corresponding executable code.

We closely follow guidance from VerilogEval(https://github.com/joshual-xlnx/verilog-eval) and notebook.ipynb.

You need to install python packages used in Genben, see file genben.py ...

If you modify the code in verilog_eval, you need to run python setup.py install after the modification to make it effective. You possibly need to modify verilog_eval/execution.py to set your own synthesis commands.

You should set your own LLM API in genben.py and DS_get_ask.py. And set LLM model in DS_get_ask.py.

os.environ["OPENAI_API_url"] = "Your own API url"
os.environ["OPENAI_API_KEY"] = "Your own API key"

1. In the first step, You can decompress the data.zip in the folder using the following designations.

unzip data.zip

2. In the second step, You can run the following command to extract the relevant topic description from the data folder in the first step and generate multiple json files according to different modalities.

python get_data.py

For example:

We used the above directive to obtain some new descriptions:

• 'descriptions_test_golden_top_code_all.jsonl'
• 'descriptions_test_golden_top_code_choice.jsonl'
• 'descriptions_test_golden_top_code_debug.jsonl'
• 'descriptions_test_golden_top_code_design.jsonl'
• 'descriptions_test_golden_top_code_text.jsonl'
• 'descriptions_test_golden_top_code_mm.jsonl'

and save them in folder save_data.

You can use DS_get_ask.py to obtain the problem description after adding disturbances; and use get_new_ds.py to obtain the JSON file of the new disturbed problem description. You should change the paths in the files properly first.

python DS_get_ask.py
python get_new_ds.py

In the last step, please be sure to modify the requests_filepath in genben.py to the newly generated JSON file from the previous step.

Run make clean-all first to clean the previous answers and test results. Use make clean-results to clean test results only.

After completing this, run the following code

python genben.py --mode all --model gpt4

• --mode: This parameter controls the type of tasks input into the LLMs. There are three available options:

  • all: Enables the input of all task types.
  • mm: Allows for multi-modal tasks.
  • text: Restricts the input to text-based tasks only.

• --model: This parameter specifies the model of the LLMs. Adjust this parameter according to the specific API of the LLMs you are using.

If you don't set parameters, these will be set to default, which you can change in the code.

After running, the results will be output from various large models that do not pass the problem, in addition to the score of the knowledge understanding and transfer questions and save them in the choice_pass.txt, and the pass@5 of the debug problems and save the results in the debug_pass.txt.

For code generation problems, the PPA, funcitonal, synatx synthesizbility results will be generated, and the PPA results will be saved in the result_pass.jsonl file, and the funcitonal and synatx results will be saved in the eval_{args.mode}_{args.model}_code.jsonl_results.jsonl. The results of Synthesizbility will be stored in eval_{args.mode}_{args.model}_code.jsonl_results_yosys_result.jsonl。

Test results are written into a table in save_data/{args.model}_test_results_{time}.csv. PPA results are written into a table in save_data/{args.model}_ppa_result_{time}.csv