The grounds of the Julia Ideson Building are the former home site of Thomas M. Bagby and his family. Thomas Bagby was Houston businessman and one of the founding members of the Houston Public Library.
The Bagby home site was at that time located across from the Martha Hermann Park (now the site of the City Hall reflecting pool).
Legend has it that Sam Houston, a close friend of the Bagby family, planted an oak tree on the property. The two large oak trees in front of the library today are said to have grown from the seedling and an acorn that came from the original tree planted by Sam Houston.