One can most accurately characterize Jim Williams' early years in Savannah as very lean. He had only one suit to his name and frequently borrowed money for routine expenses. His longtime friend, Joe Goodman, met Jim when he was just 11 years old. They were both living around Washington Square in downtown Savannah. Jim was 18 years older than Joe and became like a father figure to him. Joe's father was a merchant marine and frequently worked away from home. Jim was on a subsistence budget, so Joe's mother frequently fed Jim, as did the Saseen girls who belonged to the Saseen Bonding Company family in Savannah. Joe explained that Jim was broke for quite a long time in those early years. He bought a lot on credit and borrowed frequently.
Jim began restoring homes in 1955, when he bought three row houses on East Congress Street for very little money. Eventually he bought the whole block, rented the houses, and then sold them for a reasonable profit. Joe remembers those times vividly. Jim built his young restoration work crew from Joe and his friends that lived around Washington Square, paying them when he could.
"Want to start making some money?" Jim boomed as he enthusiastically rounded up the boys in the neighborhood. They did things like tearing down walls and putting in new plaster. They all liked him and never suspected he was gay.
Once he pocketed some money from the Congress Street restorations and rentals, Jim set his sights on the houses on nearby East St. Julian. He would put small down payments on houses to hold them until he had the funding to start work. He was not making a lot of money, but he did make an important friend, Henry Dunn, the head of Georgia State Savings Association, who funded a number of Jim's restorations.
The first house on East Julian Street that Jim set his eyes on was the Odingsells House, which was the first house of the multi-talented South Carolina native Major Charles Odingsells. Odingsells fought in the Revolutionary War, became a prosperous planter on Skidaway Island and the owner of Little Wassaw Island.
Often, Jim would live in a house after it was restored until it could be sold. In 1961, when he finished the restoration, he moved in and brought his antique shop with him.
jor restoration was the Hampton Lillibridge house on East Bryan Street. Originally, the house was going to be moved, along with another house, to Washington Square, to become the headquarters of the Port Society. During the move, one of the houses collapsed. The Historic Savannah Foundation let Jim buy and take over the restoration of the remaining Lillibridge house. Jim moved the house across the street from his home on East St. Julian and eventually bought the whole block.