Asbestos fears for former sweet factory worker