Fix the controller settings. If you're using a wheel, you'll already have looked into the advanced settings for the controls. If not though, you may be surprised with what you see. I only noticed this when I started using UDP telemetry. Out of the box, Forza Horizon 4 is set up with 10% dead zones at each end of the triggers.

forza horizon 4 drift wheel settings