there is level-based resistance, and then theres mob-specific resistances.
all monsters of same level have the same level-resistance, this is relative to the players level when casting. level-resistance cannot be reduced in any way.
mob-specific resistance vary from monster to monster. Most monsters dont have any at all. raid bosses have varying amounts of specific resistances, and things like curse of elements, and spell penetration will reduce this.
for a lvl 60 character casting on a lvl 63 boss, the level-resistance equates to about 6% average loss of damage. binary spells (frostbolt for example) ignore level-resistance, so it is possible to end up with 1% chance to miss when fully hit-capped.
A fireball (non-binary spell) will end up with 1% miss chance when capped, but then a portion of the damage (6%) will be lost through partial resists, even if the boss has 0 resistance (it still has level resistance).
theres already several calculators lying around, i can recommend this one Spelldamage Calculators by Ayz