The Ginsters and West Cornwall Pasty Co owner said Covid’s impact on buying habits had led to a “very difficult and challenging”