When Elemental Hit chooses fire - which it always will - it will then deal more damage.
This includes the flat damage from the Gem. 50% of all Lightning and 50% of all Cold Damage are converted to Fire. Avatar of Fire granted by the Unique Amulet - Xoph's Blood increases your damage by converting it to Fire Damage, while you focus on increasing your Fire Penetration. The build presented here is all about using an Elemental Hit, focused on dealing Fire Damage using a Bow. Use Combat Focus - Crimson and Viridian Jewels so that your Elemental Hit can only 'pick' Fire Damage.