The construction of this severe and forbidding[3] building was commissioned in 1458 by the Florentine banker Luca Pitti (1398–1472), a principal supporter and friend of Cosimo de' Medici. The early history of the Palazzo Pitti is a mixture of fact and myth. Pitti is alleged to have instructed that the windows be larger than the entrance of the Palazzo Medici. The 16th-century art historian Giorgio Vasari proposed that Brunelleschi was the palazzo's architect, and that his pupil Luca Fancelli was merely his assistant in the task, but today it is Fancelli who is generally credited.[4] Besides obvious differences from the elder architect's style, Brunelleschi died 12 years before construction of the palazzo began. The design and fenestration suggest that the unknown architect was more experienced in utilitarian domestic architecture than in the humanist rules defined by Alberti in his book De Re Aedificatoria.[5]
Though impressive, the original palazzo would have been no rival to the Florentine Medici residences in terms of either size or content. Whoever the architect of the Palazzo Pitti was, he was moving against the contemporary flow of fashion. The rusticatedstonework gives the palazzo a severe and powerful atmosphere, reinforced by the three-times-repeated series of seven arch-headed apertures, reminiscent of a Roman aqueduct. The Roman-style architecture appealed to the Florentine love of the new style all'antica. This original design has withstood the test of time: the repetitive formula of the façade was continued during the subsequent additions to the palazzo, and its influence can be seen in numerous 16th-century imitations and 19th-century revivals.[5] Work stopped after Pitti suffered financial losses following the death of Cosimo de' Medici in 1464. Luca Pitti died in 1472 with the building unfinished.[6]

The building was sold in 1549 to Eleonora di Toledo. Raised at the luxurious court of Naples, Eleonora was the wife of Cosimo I de' Medici of Tuscany, later the Grand Duke.[4] On moving into the palace, Cosimo had Vasari enlarge the structure to fit his tastes; the palace was more than doubled by the addition of a new block along the rear. Vasari also built the Vasari Corridor, an above-ground walkway from Cosimo's old palace and the seat of government, the Palazzo Vecchio, through the Uffizi, above the Ponte Vecchio to the Palazzo Pitti.[7] This enabled the Grand Duke and his family to move easily and safely from their official residence to the Palazzo Pitti. Initially the Palazzo Pitti was used mostly for lodging official guests and for occasional functions of the court, while the Medicis' principal residence remained the Palazzo Vecchio. It was not until the reign of Eleonora's son Francesco I and his wife Johanna of Austria that the palazzo was occupied on a permanent basis and became home to the Medicis' art collection.[8]
Land on the Boboli hill at the rear of the palazzo was acquired in order to create a large formal park and gardens, today known as the Boboli Gardens.[4] The landscape architect employed for this was the Medici court artist Niccolò Tribolo, who died the following year; he was quickly succeeded by Bartolommeo Ammanati. The original design of the gardens centred on an amphitheatre, behind the corps de logis of the palazzo.[5] The first play recorded as performed there was Andria by Terence in 1476. It was followed by many classically inspired plays of Florentine playwrights such as Giovan Battista Cini. Performed for the amusement of the cultivated Medici court, they featured elaborate sets designed by the court architect Baldassarre Lanci.[9]