Fishermen in smaller fleets say the introduction of tariffs would hit the industry hard