Tracking down a 25% Regression on LLVM RISC-V

Tracking down a 25% Regression on LLVM RISC-V

Similar to the previous post, this post covers my analysis of a benchmark on RISC-V targets. Unlike the previous post, I was able to land a patch to eliminate the performance gap to GCC (for this benchmark)!

TLDR

A recent LLVM commit improved isKnownExactCastIntToFP to fold fpext(sitofp x to float) to double into a direct uitofp x to double cast, but this inadvertently broke a downstream narrowing optimization in visitFPTrunc that relied on the fpext to narrow a double to float, causing a ~24% performance regression on RISC-V targets, where fdiv.d (33 cycle latency) was emitted instead of fdiv.s (19 cycle latency).

My fix extends getMinimumFPType with range analysis to recognize that fptrunc(uitofp x double) to float can be reduced to uitofp x to float, restoring the narrowing optimization.

I was looking at Igalia’s site comparing the performance of LLVM to GCC on RISCV targets, and I noticed this particular benchmark. As shown in the image below, LLVM requires about ~8% more cycles than GCC for that specific benchmark on the SiFive P550 CPU.

I have included snippets of the relevant basic block assembly. Practically all the cycles were spent on the assembly below. From the two assembly, it wasn’t immediately obvious to me why GCC was doing better. They seemed almost identical, and if anything, LLVM was able to optimize the branch logic here. The big difference I did notice was that LLVM was doing an fdiv.d, a division with double precision floating point or an f64.

This did seem promising so I decided to run llvm-mca on the source code to get a better idea of what is happening. Note that I am using llvm-mca built from source that was a couple of days old from upstream. From the analysis done by llvm-mca, I noticed that the fdiv.d instruction made no appearance in the loop. Although I did not show it above, both GCC and LLVM contained an fdiv.d instruction in a later basic block but, this was outside the main loop and therefore not relevant to the performance difference.

Info: llvm-mca

llvm-mca is a tool within the llvm suite that can be used to statically measure the performance of machine code for a specific CPU.

$LLVM_BUILD_DIR/bin/llvm-mca -mtriple=riscv64 -mcpu=sifive-p550 pi.s

Around the area where there should have been an fdiv.d, there were two fdiv.s like GCC. From this, I concluded that the fdiv.d instruction in the loop must have been a recent regression. LLVM used to be able to narrow the double to a float, but the latest builds can no longer convert the double into a float. I confirmed this was indeed a regression by comparing LLVM to a prior build for the same CPU and benchmark.

If you are wondering as to why the ordering of the assembly looks different, it’s because the target CPU, the SiFive P550, is an out of order CPU. Unlike in-order CPUs, this target can execute instructions in an order that can produce a higher throughput. Since the double division has a significantly higher latency, the CPU is trying to dispatch other instructions to ‘hide’ this latency. The fcvt.s.d instruction that consumes the value of the fdiv.d would need to wait for 33 cycles, so perhaps the CPU made the decision to schedule instructions between them.