My oldest "ball" related item is this book, acquired from Barry Sloate. It dates from about 1845-1850. The picture clearly shows what appears to be a cricket game, but the term "ball" was apparently in use by this time.
