Grower bodies have warned the lack of clarity on post-Brexit trading and labour provision could lead to price rises and shortages