On May 22, 1872, the Amnesty Act restored civil rights to Southerners (except for 500). From the article:
"The End of Reconstruction
As time passed, it became more and more obvious that the problems of the South were not being solved by harsh laws and continuing rancor against former Confederates. In May 1872, Congress passed a general Amnesty Act, restoring full political rights to all but about 500 Confederate sympathizers.
Gradually Southern states began electing members of the Democratic Party into office, ousting so-called carpetbagger governments and intimidating blacks from voting or attempting to hold public office. By 1876 the Republicans remained in power in only three Southern states. As part of the bargaining that resolved the disputed presidential elections that year in favor of Rutherford B. Hayes, the Republicans promised to end Radical Reconstruction, thereby leaving most of the South in the hands of the Democratic Party. In 1877 Hayes withdrew the remaining government troops, tacitly abandoning federal responsibility for enforcing blacks' civil rights.
The South was still a region devastated by war, burdened by debt caused by misgovernment, and demoralized by a decade of racial warfare. Unfortunately, the pendulum of national racial policy swung from one extreme to the other. Whereas formerly it had supported harsh penalties against Southern white leaders, it now tolerated new and humiliating kinds of discrimination against blacks. The last quarter of the 19th century saw a profusion of "Jim Crow" laws in Southern states that segregated public schools, forbade or limited black access to many public facilities, such as parks, restaurants and hotels, and denied most blacks the right to vote by imposing poll taxes and arbitrary literacy tests.
In contrast with the moral clarity and high drama of the Civil War, historians have tended to judge Reconstruction harshly, as a murky period of political conflict, corruption and regression. Slaves were granted their freedom, but not equality. The North completely failed to address the economic needs of the freedmen. Efforts such as the Freedmen's Bureau proved inadequate to the desperate needs of former slaves for institutions that could provide them with political and economic opportunity, or simply protect them from violence and intimidation. Indeed, federal Army officers and agents of the Freedmen's Bureau were often racists themselves. Blacks were dependent on these Northern whites to protect them from white Southerners, who, united into organizations such as the Ku Klux Klan, intimidated blacks and prevented them from exercising their rights. Without economic resources of their own, many Southern blacks were forced to become tenant farmers on land owned by their former masters, caught in a cycle of poverty that would continue well into the 20th century.
Reconstruction-era governments did make genuine gains in rebuilding Southern states devastated by the war, and in expanding public services, notably in establishing tax-supported, free public schools for blacks and whites. However, recalcitrant Southerners seized upon instances of corruption (hardly unique to the South in this era) and exploited them to bring down radical regimes. The failure of Reconstruction meant that the struggle of African Americans for equality and freedom was deferred until the 20th century -- when it would become a national, and not a Southern issue."