From what I know, things were really terrible for them on Sylvarant when they were doing the pilgrimage. Course, most of us don't talk about most of the things that happened for our own reasons, regardless of your reasons for chasing Lloyd at the start.