Can LLMs Generate FPGA-Ready Verilog?
FPGABench tests generated hardware designs beyond simulation, measuring whether they can actually synthesise, meet physical FPGA constraints, and optimise for a design objective.
Feasibility vs. Design Quality Frontier
Visualizing model capability. The Pareto frontier curve connects models achieving the optimal balance between functional feasibility and compilation/PPA quality.
FPGABench Leaderboard
Sort by columns to find the most cost-effective, high-quality, or functionally correct models.
| Rank ↑ | Model Name ↕ | Developer ↕ | License ↕ | Feasible@1 ↕ | MOQ@1 ↕ | Details |
|---|
What is FPGABench?
Unlike general code generation benchmarks (e.g., HumanEval) that only verify functional execution, FPGA Verilog development is tightly bound to hardware constraints. A design that runs correctly in simulation can still fail synthesis, exceed resource budgets, or run too slowly on a real chip.
FPGABench introduces 52 diverse, hand-crafted hardware design problems ranging from basic arithmetic blocks (multipliers, ALUs) to complex sequential circuits (FIFO controllers, SPI masters, AES pipelines). It automatically synthesizes generated code to evaluate post-routing hardware metrics.
Key Metrics Defined
A candidate design $d$ is feasible for problem $p$ iff it is functionally correct, synthesisable, and meets all constraints. $\text{Feasible@}K$ is then the fraction of problems where model $m$ produces a feasible design in $K$ attempts:
For a feasible design $d$, the quality ratio $q_p(d)$ measures performance relative to the reference design ($q_p(d) > 1$ improves on the reference, while $q_p(d) < 1$ underperforms). $\text{MOQ@}K$ is the geometric mean of the best quality ratio found within $K$ attempts:
Table 4.3: Scoring examples for the 7-Sample Median Denoising Filter
A detailed illustration of candidate evaluation for a specific hardware design target under Feasible@K and MOQ@K metrics.
| Candidate ID | Functionally Correct | Synthesizable | Meets Constraints | LUT Usage | Feasible | Quality ratio ($q_p(d)$) |
|---|---|---|---|---|---|---|
| Candidate 1 (Optimal) | ✅ Yes | ✅ Yes | ✅ Yes | 80 (Reference: 100) | Yes | 1.25 (Improves PPA) |
| Candidate 2 (Suboptimal) | ✅ Yes | ✅ Yes | ✅ Yes | 120 (Reference: 100) | Yes | 0.83 (Underperforms) |
| Candidate 3 (Fails Constraint) | ✅ Yes | ✅ Yes | ❌ No (Timing Violation) | 70 (Reference: 100) | No | — (Infeasible) |
| Candidate 4 (Fails Function) | ❌ No | ✅ Yes | — | — | No | — (Infeasible) |
Submit Your Model Results
We encourage researchers and developers to run FPGABench on new models. To add your model to the official leaderboard, please follow the steps below:
git clone https://github.com/fpgabench/fpgabench.git
python run_eval.py --model your-model-name --eda_tool vivado
results/your_model.json.