His disagreement with Francesco Maria Della Rovere
Castiglione, although he had been carrying out diplomatic duties for a couple of years on behalf of Federico Gonzaga, only in July 1521 did he ask the Duke of Urbino to release him from any form of dependency, and, to obtain this, solicited the intercession of Elisabetta Gonzaga. In the light of the new alliance between Mantua and Rome and of his position with Leone X, he felt he was no longer in a position to defend the interests of Francesco Maria Della Rovere.
At the end of 1521, with Leone X dead, Francesco Maria Della Rovere informed Castiglione that, after having reoccupied the state of Urbino, as a condition for maintaining his dominion over Pesaro he had to promise the restitution of the castle and estate of Novilara, that he had given to him as a gift in 1513. Deprived without further ado of his fiefdom, as if he had been an enemy and a traitor, Baldassarre, on 2nd January 1522, wrote a grave and vehement letter of protest to the Duke, which was followed shortly by others, that mixed pride, pique and a desire to have his rights recognised. Furthermore, Baldassarre, did not hesitate to lay claim to services rendered and that could have been rendered with the Pope, benefiting Francesco Maria. Thus a solomonic, peaceful and accommodating solution was reached: Castiglione kept the feudal title of Count of Novilara, but effective ownership of the surrounding lands returned to the Duke of Urbino.
With Baldassarre dead, his mother Aloisia Gonzaga considered the possibility of asking for the fiefdom again for her nephew Camillo, but she received nothing more form Francesco Maria Della Rovere than generic and evasive promises. One can deduce from this the resentment that the Duke had matured towards Castiglione, guilty, in his eyes, of not having assisted him adequately, right from 1517, in his war to re-conquer the state, and worthy, therefore, of being deprived of all the benefits that had originally compensated his merits.

