Is your buffer size set to 128 samples or less? Open up the I/O Window and pick the buffer size that works best on your system. We recommend a buffer size no more than 256 samples.
How far are you from your speakers? Every additional foot of space adds approximately 1 millisecond of latency between when you hit the drum and when you hear the sound, which adds up. Monitoring with headphones eliminates this problem.
Close all unnecessary applications on your computer. Running several large applications can cause latency.
Ensure that you have the correct and most up to date software installed for your audio interface.
It might not be computational latency at all! Make sure that all of the dead-space is removed from the front of your samples. See trimming.