In his diary Jack Jackman mentions that the country was so bare of foodstuffs that he and his fellows didn't eat for a week in their retreat from Perryville. Plus this was at the end of the summer and in the second year of a drought so yes, Buell was right.
But what did him in with Helleck and the rest of the high command, at least as I see it, was that Buell had been making excusses since he replaced Sherman almost a year eariler. In other words he'd cried wolf one too many times.