I will admit that I consulted my good friend, Google, for the answer. But it was still kind of hard to find it, I had to do a few different searches and dug around till I found it.
Date of the game was April 18, 1823. In addition to the first HR in yankee stadium (a 3-run blast which Ruth hit in the 3rd inning), the Babe also had the first error - a dropped fly ball in the 5th inning. (edit: billyb, you beat me to it, lol...)
Box Score:
http://www.baseball-reference.com/bo...92304180.shtml