The first way I found to do this was by adding the GS_PROG environment variable (add the path to the .exe file), but I’ve got the portable version of GIMP and it seemed to not be checking the environment variables. After much internet searching I found this.

The solution was to copy the executable files from C:\Program Files\gs\gs8.54\bin into GIMPPortable\App\gimp\bin

Some day google will index this page, and I hope others find this as easily as I should have been able to the first time.