Grierson was born in Pittsburgh, Pennsylvania, the youngest of 5 siblings. He became afraid of horses when at age 8, he was kicked and nearly killed by a horse. In 1851, he became music teacher and band leader in Jacksonville, Illinois, and then entered business. He married Alice Kirk of Youngstown on
September 24, 1854. The couple had 7 children, 4 of whom survived to adulthood.
When the Civil War began, Grierson enlisted in the U.S. Army as a private. He was assigned as a volunteer aide-de-camp to Major Gen. Benjamin M. Prentiss. Promoted to major, he joined the 6th Illinois Cavalry and was promoted to Colonel of that regiment on April 12, 1862. His regiment was engaged in a number of small skirmishes and raids on railroads and facilities in West Tennessee and northern Mississippi that spring and summer. In November, he became a brigade commander in the Cavalry Division of the Army of the Tennessee. In December, he participated in the pursuit of Confederate Lieutenant Gen. Earl Van Dorn after his Holly Springs Raid against the supply lines of Lieutenant Gen. Ulysses S. Grant.
Grierson became famous for leading the cavalry raid in the Grierson's Raid in 1863, a major diversionary thrust deep into the Confederacy. The raid was ordered by Grant as part of his Vicksburg Campaign. One of the more important results of the raid was that he diverged the attention of the Confederate defenders of Vicksburg away from Grant's main thrust. It also showed that showed that Union troops could operate in Confederate areas without a supply line. He was promoted to Brigadier General of volunteers in June. According to Grant, the raid had "taken the heart out of Mississippi." In 1864 he was assigned to the Cavalry Corps of the Army of the Mississippi and in 1865, he took part in the Mobile Campaign .
After the war, Grierson decided to remain in the regular army after the war and received the rank of Colonel. His lack of West Point credentials made him suspect to many fellow officers. On March 2, 1867, he received a brevet promotion to the rank of major general in the Regular Army for his famous raid. He organized the 10th U.S. Cavalry, 1 of 2 mounted regiments composed of black enlisted men and white officers, called the Buffalo Soldiers. This assignment also made him unpopular with other officers, including his superior, Major Gen. Philip Henry Sheridan, because of his support for and trust in his troops. His sympathy and courtesy to Native American tribes also led to questions about his judgment. He commanded the Department of Arizona and Districts of New Mexico and Indian Territory. He retired from the military in 1890. He became one of the few civilians who attained the rank of Brigadier General in the Regular army.
Alice, Grierson's first wife, died on August 14, 1888. He later married Lillian Atwood King, a widow, July 28, 1897. In 1907, Grierson suffered a debilitating stroke. In 1911, he was buried in Jacksonville Cemetery (East Cemetery), Jacksonville, Illinois.