The chairman of Chelsea's Conference League semi-final opponents has resigned after being caught up in a racism storm.