The War Memorial Veterans Building, located beside the War Memorial Opera House in the Civic Center of San Francisco was authorized by the people of San Francisco in 1920. The building was constructed for the veterans, partly with funds contributed by the members of the American Legion, in recognition and appreciation for their valiant efforts during the First World War. The Trust Agreement of 1921 created the Board of Trustees appointed by the mayor and the War Memorial Commission whose members are chosen by the individual posts of the American Legion. The trustees are responsible for the general business of the two War Memorial buildings and the commission is responsible for the veterans interests in the Veterans Building.

The Veterans Building was constructed during the years 1933-34 and is built of gray stone embellished in the interior with marble throughout its four floors. Various veterans groups including posts of the American Legion presently occupy the first and second floors. The third floor temporarily houses the Municipal Transit Agency headquarters. Since 1977, the Hearbst Theater was transferred to The Performing Arts Center and the Green Room on the second floor was relinquished by the War Memorial Commission. Part of the basement is rented to the Museum of Modern Art.