They are small cylinders, so without gloves lots of half prints will be left. Fingerprints are so similar that that is unlikely to be usable to find the loader without a ton of effort. It is very useful to tie a suspect to the weapon after they have been found through other means, however.

A smart criminal should always wear gloves when interacting with anything that may be collected as evidence.