The Doolittle Guest House was built in 1872 by the George Doolittle family. It is the oldest house on the block. The facade is contructed of rusticated sandstone, quarried in Potsdam, New York, and includes an arched-top transom; stained- and leaded-glass windows that contain both American Aesthetic and Neoclassical elements; and a second-story corbeled bay window with a commanding view of East Capitol Street.

The distinctive Richardsonian facade, front and back rooms, and coffered ceilings and wainscoting were part of an 1887 addition completed during the ownership of Seymour W. Tullock. The architect of the addition was Robert Stead, who, along with Frank Miles Day and Wilson Eyre, was instrumental in securing and restoring the Octagon House for the American Institute of Architects.

Another prominent owner of the house was Mary E. French, widow of B.B. French, Commissioner of Public Buildings in Washington, DC (the equivalent of Mayor), during the Pierce (1850-1853), Lincoln (1861-1865) and first two years of the Johnson (1865-1867) administrations. Ms. French was also the aunt of Daniel Chester French, sculptor for the Lincoln Memorial. The French family owned the house from 1904 until 1911.

Inside the house, distinguishing features include American chestnut wainscoting in the foyer, stairs, hallway and dining room and American walnut coffered ceilings in the foyer and dining room. The house's antique furnishings are from the Federalist and Centennial periods.