Let's say your site is based around a specific product or product line. Should you have Adsense on the site too? Chances are, the ads shown will be very similar to the items you have for sale. So will that cause you to get fewer sales?
Maybe you could advertise your Adsense on certain pages only, so it doesn't compete with your affiliate products. For these types of sites, Adsense could be perfect to place within posts that don't directly relate to your affiliate product. For instance, you would not want to put Adsense on your affiliate review posts.