In Hamlet, why does Ophelia return the letter and presents Hamlet gave her?

Ophelia does this at the instruction of her father, Polonius. He believes Hamlet is not serious with her and is playing around. Polonius also tells Ophelia she is too young and inexperienced to be involved in (what he believes to be) a risky venture, as he tells her during their conversation in Act II, Scene 3:

OPHELIA
He hath, my lord, of late made many tenders
Of his affection to me.

LORD POLONIUS
Affection! pooh! you speak like a green girl,
Unsifted in such perilous circumstance.
Do you believe his tenders, as you call them?

Furthermore, he contends that Hamlet, being a prince, is far beyond Ophelia's status, as she is not royalty. Being the daughter of the king's adviser does not naturally mean she can have a relationship with the prince. He commands her to have no further relations with him:

I would not, in plain terms, from this time forth,
Have you so slander any moment leisure,
As to give words or talk with the Lord Hamlet.
Look to't, I charge you: come your ways.

Polonius also believes Hamlet is going mad and seems to be concerned about his seeming obsession with Ophelia. When Ophelia informs him of Hamlet's verbal declarations of love, the gifts he gave her, and the affectionate letters he wrote her, Polonius insists Ophelia return them.

In Act II, Scene 2, Polonius reports Hamlet's strange behavior towards Ophelia to Claudius and Gertrude, telling them he disciplined his daughter by telling her, 

'Lord Hamlet is a prince, out of thy star;
This must not be:' and then I precepts gave her,
That she should lock herself from his resort.

When Polonius later speaks to Hamlet, he takes particular note of his constant references to Ophelia and assumes Hamlet is obsessed with her and mentally unstable. 

In Act III, Scene 1, Ophelia confronts Hamlet and tells him that she wishes to return all the "remembrances" he had given her. Hamlet denies that he had given her anything. During their conversation, the prince gets quite upset and insults Ophelia, telling her: "get thee to a nunnery," for she should not breed sinners. He accuses her of lying. Ophelia is obviously unnerved about his ranting and calls upon the powers of heaven to restore him. Hamlet storms off.

Polonius and Claudius have been eavesdropping on the conversation and then discuss Hamlet's behavior. They agree that all is not well with the young prince. Claudius undertakes to send him to England, while Polonius advises that the king should ask Gertrude to consult with her son. If Gertrude cannot make any headway, Polonius says Hamlet should be sent to England. Claudius ends the scene by stating,

It shall be so:
Madness in great ones must not unwatch'd go.