FIFTH AMENDMENT AGREEMENT To Pledge and Security Agreement and Irrevocable Proxy

Exhibit 10.36

EXECUTION COPY

FIFTH AMENDMENT AGREEMENT

To Pledge and Security Agreement and Irrevocable Proxy

Dated as of June 30, 2009

by and among

RFC ASSET HOLDINGS II, LLC,

PASSIVE ASSET TRANSACTIONS, LLC

AND CERTAIN

AFFILIATES THEREOF FROM

TIME TO TIME PARTY HERETO,

as Grantors,

and

GMAC LLC,

as Lender Agent

This FIFTH AMENDMENT AGREEMENT (this “ Agreement ”), dated as of June 30, 2009 (the “ Amendment Effective Date ”), is by and among RFC Asset Holdings II, LLC, a Delaware limited liability company (“ RAHI ”), and Passive Asset Transactions, LLC, a Delaware limited liability company (“ PATI ”; and together with RAHI, each a “ Borrower ” and collectively, the “ Borrowers ”); Residential Capital, LLC, a Delaware limited liability company (“ ResCap ”), Residential Funding Company, LLC, a Delaware limited liability company (“ RFC ”), and GMAC Mortgage, LLC, a Delaware limited liability company (“ GMAC Mortgage ”; and together with ResCap and RFC, each herein a “ Guarantor ” and collectively, the “ Guarantors ”), and the other parties hereto as Grantors (each, together with each Borrower and each Guarantor, a “ Grantor ” and collectively, the “ Grantors ”); and GMAC LLC, a Delaware limited liability company, as Lender Agent for the Lender Parties.

Reference is hereby made to the Pledge and Security Agreement and Irrevocable Proxy dated as of November 20, 2008 among the Grantors and the Lender Agent (as amended and modified through the date hereof, the “ Security Agreement ”).

RECITALS

1. Each of the parties hereto is a party to the Security Agreement.

2. The parties hereto desire to make certain amendments to the Security Agreement.

3. Each of the parties hereto, by its signature hereto, hereby acknowledges, consents and agrees to the changes set forth herein.

4. In consideration of the promises and mutual agreements herein contained and for good and valuable consideration, the receipt and sufficiency of which is hereby acknowledged, the parties hereto agree as follows:

ARTICLE I

DEFINED TERMS

SECTION 1.1 Capitalized terms used herein and not otherwise defined shall have the meaning set forth in the Security Agreement.

ARTICLE II

AMENDMENTS TO THE SECURITY AGREEMENT

SECTION 2.1 Amendments to Section 1 .

(a) The following definitions in Section 1 to the Security Agreement are hereby amended and restated in their entirety to read as follows:

First Savings Warehouse Note means that Promissory Note, dated as of April 1, 2005 and that Promissory Note dated as of July 1, 2009 and issued by First Savings Mortgage Corporation in favor of RFC, as the same may be amended, supplemented, restated or otherwise modified from time to time, and including any notes given in substitution or replacement therefor.

(b) The following definitions are hereby added to Section 1 to the Security Agreement:

MSR Documents means (a) the MSR Loan Agreement, (b) the “Notes, “ the “Guarantee,” the “Account Control Agreements,” and the “Servicing Contracts” (each as defined in the MSR Loan Agreement) and (c) all notices, certificates, financing statements and other documents to be executed and delivered by RFC, GMAC Mortgage or ResCap in connection with the transactions contemplated by the MSR Loan Agreement.

November Documents means (a) the Loan Agreement, the Notes, this Agreement, the Account Control Agreements, the GSAP Indenture Transaction Documents and the Flume No. 8 Security Documents, (b) after the Custodial Transfer Date, the Master Custody Agreement, (c) any document designated as a “Security Document” in any Collateral Addition Designation Notice, (d) all of the security agreements, pledges, collateral assignments, mortgages, deeds of trust, trust deeds or other instruments evidencing or creating or purporting to create any security interests in favor of the Lender Agent for its benefit and for the benefit of the Lender Parties, (e) the Contribution Agreements and (f) all notices, certificates, financing statements, agreements and other documents to be executed and delivered by RAHI, PATI, ResCap, RFC or GMAC Mortgage pursuant to the foregoing or otherwise in connection with the Loan Agreement or the extension of financing by the Lenders contemplated thereunder.

Warehouse Loans means, as the context may require, loans made by RFC pursuant to a Warehouse Agreement to the borrower under such agreement and/or any mortgage loans or other loans or assets purchased by RFC pursuant to the First Savings Warehouse Agreement.

(c) The definitions of “Permitted Liens”, “Secured Parties” and “Credit Agent” are hereby deleted in their entirety from Section 1 to the Security Agreement.

SECTION 2.2 Amendments to Schedule IV .

(a) Exhibit A to Schedule IV of the Fourth Security Agreement is hereby amended by inserting the following:

(b) Exhibit C to Schedule IV of the Security Agreement is hereby amended by amending and restating the third Pledged Note listed therein as follows:

(c) Exhibit D to Schedule IV of the Security Agreement is hereby amended by inserting the following: