SIXTH AMENDMENT To Loan Agreement Dated as of May 19, 2009 by and among

Exhibit 10.6

EXECUTION COPY

SIXTH AMENDMENT

To Loan Agreement

Dated as of May 19, 2009

by and among

PASSIVE ASSET TRANSACTIONS, LLC,

as Borrower,

RFC ASSET HOLDINGS II, LLC,

as Borrower,

RESIDENTIAL FUNDING COMPANY, LLC,

as Guarantor,

GMAC MORTGAGE, LLC,

as Guarantor,

RESIDENTIAL CAPITAL, LLC

as Guarantor,

Certain Affiliates of the Borrowers and Guarantors

party hereto as Obligors,

GMAC LLC,

as Initial Lender and as Lender Agent

and

Certain Other Financial Institutions and Persons from

time to time party hereto as Lenders

This SIXTH AMENDMENT (this “ Agreement ”) dated as of May 19, 2009 (the “ Amendment Effective Date ”), is by and among Passive Asset Transactions, LLC, a Delaware limited liability company (“ PATI ”), RFC Asset Holdings II, LLC, a Delaware limited liability company (“ RAHI ” and, together with PATI, each a “ Borrower ” and collectively, the “ Borrowers ”), Residential Funding Company, LLC, a Delaware limited liability company (“ RFC ”), Residential Capital, LLC, a Delaware limited liability company (“ ResCap ”), GMAC Mortgage, LLC, a Delaware limited liability company (“ GMAC Mortgage ”, and together with RFC and ResCap, each a “ Guarantor ” and collectively, the “ Guarantors ”), the various other parties signatory hereto as obligors (the “ Obligors ”) GMAC LLC, a Delaware limited liability company (the “ Initial Lender ”), the financial institutions and other Persons that are or may from time to time become parties hereto as Lenders (together with the Initial Lender and their respective successors and assigns, each a “ Lender ” and collectively, the “ Lenders ”) and GMAC LLC, a Delaware limited liability company, as agent for the Lenders (in such capacity together with its successors and assigns in such capacity, the “ Lender Agent ”).

Reference is hereby made to the Loan Agreement, dated as of November 20, 2008 among the Borrowers, the Guarantors, the Lenders and the Lender Agent (as amended and modified through the date hereof, the “ Loan Agreement ”).

RECITALS

1. Each of the parties hereto is a party to the Loan Agreement.

2. The parties hereto desire to make certain amendments to the Loan Agreement.

3. Each of the parties hereto, by its signature hereto, hereby acknowledges, consents and agrees to the matters set forth herein.

4. In consideration of the premises and mutual agreements herein contained and for good and valuable consideration, the receipt and sufficiency of which is hereby acknowledged, the parties hereto agree as follows:

ARTICLE I

DEFINED TERMS

SECTION 1.1 Capitalized terms used herein and not otherwise defined shall have the meaning set forth in the Loan Agreement.

ARTICLE II

AMENDMENTS TO THE LOAN AGREEMENT

SECTION 2.1 Amendments to the Loan Agreement . Each of the parties hereto hereby consents and agrees that the Loan Agreement shall be amended as of the Amendment Effective Date as follows:

(a) Section 2.04(b) of the Loan Agreement is hereby amended and restated to read as follows:

“(b) After the Initial Funding Date, the Borrowers shall deliver to the Lender Agent an updated Monthly Collateral Report no less frequently than once per calendar month and no later than the eleventh Business Day of each calendar month (commencing with December 2008). In addition, on each REO Supplement Date (commencing with June, 2009) the Borrowers shall deliver to the Lender an REO Supplemental Report. For purposes of preparing each Collateral Value Report, the Borrowers shall calculate the Collateral Value of the Qualifying Collateral in accordance with Schedule 2.04 .”

(b) Section 2.04(c) of the Loan Agreement is hereby amended by replacing “the Flume No. 8 Note” in the first sentence of Section 2.04(c) with “a European Note”.

(c) Section 2.08(d) of the Loan Agreement is hereby amended and restated in its entirety to read as follows:

“[Reserved.]”

(d) Section 4.01 of the Loan Agreement is hereby amended by deleting the phrase “(other than Collections with respect to the Flume 8 Note and the related Supporting Assets)” from the first sentence of Section 4.01 .

(e) Section 4.02 of the Loan Agreement is hereby amended and restated in its entirety to read as follows:

“Section 4.02. Collections with Respect to European Collateral . The Borrowers shall cause to be established the European SPV Accounts. Any Collections with respect to the Supporting Assets for a European Note shall be deposited in the applicable European SPV Accounts in accordance with the terms of the applicable European Security Documents. Any Collections with respect to a European Note shall be deposited by the Obligors in a Collection Account no later than the Applicable Deposit Date; it being understood that the Obligors may deposit such Collections in the applicable European Hedging Account prior to deposit in a Collection Account on or prior to the Applicable Deposit Date.”

(f) Section 7.01(m) of the Loan Agreement is hereby amended and restated in its entirety to read as follows:

“(m) Custodial Procedures . On or before the Custodial Transfer Date, the Borrowers and Obligors will enter into one or more custody agreements (as the same may be amended, supplemented, modified or restated from time to time, the “ Custody Agreements ”) governing the custody of certain documentation for Collateral consisting of US Mortgage Loans, in form and substance satisfactory to the Lender Agent, which Custody Agreements shall appoint Wells Fargo Bank National Association and Ally Bank as custodians. The Obligors will use commercially reasonable efforts to cause such custodians to have obtained and verified their possession of all notes, and other documents described in the Custody Agreements as being deliverable to such custodians no later than August 19, 2009, and any US Mortgage Loans for which

the applicable custodian has not verified possession of the related Mortgage Note by August 20, 2009 shall no longer constitute Eligible Assets after such applicable date unless otherwise agreed in writing by the Lender Agent. The Obligors and the Lender shall discuss in good faith how the items included on each custodian’s exception report shall be addressed prior to September 2, 2009. It is understood and agreed that the Lender Agent may adjust the Specified Percentage of all or a portion of the US Mortgage Loans in response to the exceptions described therein.”

(g) Section 7.01(q) of the Loan Agreement is hereby amended and restated in its entirety to read as follows:

“(q) No later than May 29, 2009, the Obligors and Lender Agent shall agree upon a structure to protect the Lender Agent’s and the Omnibus Agent’s interest in REO Property resulting from the foreclosure of US Mortgage Loans. It is understood and agreed that the Lender Agent may, based on the structure agreed upon or the failure to agree upon a structure, adjust the Specified Percentage for all or a portion of the US Mortgage Loans.”

(h) Section 7.01(v) of the Loan Agreement is hereby amended and restated in its entirety to read as follows:

“(v) Instructions under Flume No. 8 Agreements and GX II Agreements . ResCap shall, at the specific written direction of the Lender Agent, give (or refrain from giving) any instructions permitted to be given under the European Security Documents.”

(i) The first sentence of Section 7.01(w) of the Loan Agreement is hereby replaced in its entirety with the following:

“The Obligors shall provide not less than the Specified Days prior written notice to the Lender Agent of any proposed Approved Additional Collateral which they request to be added to the Collateral, and shall deliver such documents, agreements, schedules and other information as the Lender Agent shall request in connection with any such proposed Approved Additional Collateral including, without limitation, all information with respect to any equity in joint ventures or other Assets acquired by Equity Investment I. The Lender Agent shall act in good faith to discuss any such request from the Obligors.

If any proposed Approved Additional Collateral consists of US Mortgage Loans, the Obligors shall deliver to the Lender Agent a schedule of the proposed Approved Additional Collateral, which schedule shall include loan data, with the same detail and in the same format, as the data file delivered to the Lender Agent in connection with the first Group A Loans included in the Collateral (or such other data, detail or format as the Lender Agent and the Obligors shall mutually agree), and in any event sufficient data to identify each Mortgage Loan thereon and shall include both (i) US Mortgage Loans proposed to be added to the Group A Loans and the Group B Loans and (ii) Group A Loans and Group B Loans

included in the Collateral at the time such schedule is delivered. In connection with the delivery of the Collateral Addition Designation Notice, the Obligors shall (i) deliver a Mortgage Schedule to the Lender Agent and the Omnibus Agent and a final data file in the same format and containing the same information as the original data file, but containing only those loans included on the Mortgage Schedule and those US Mortgage Loans included in the Collateral at the time such Mortgage Schedule is delivered, (ii) prepare and file a UCC-3 financing statement adding a description of the Mortgage Schedule to the financing statements outstanding at such time, which shall be in form and substance acceptable to the Lender Agent and its counsel and (iii) deliver any lien releases and related UCC-3 financing statements required to release any outstanding liens on such Mortgage Loans, which shall be in form and substance acceptable to the Lender Agent and its counsel. Each Obligor shall promptly mark its books and records to indicate that all Mortgage Loans included on a Mortgage Schedule have been pledged to the Lender Agent for so long as such Mortgage Loan constitutes Collateral.”

(j) Section 7.02(k) of the Loan Agreement is hereby amended and restated as follows:

“(k) consummate a Collateral Disposition unless (i) the related Collateral Disposition Proceeds are deposited as Collections in accordance with Article IV , (ii) the Lender Agent shall have consented to such Collateral Disposition unless such Collateral Disposition consists of (A) the Transfer of whole Mortgage Loans in the ordinary course of business and the Collateral Disposition Proceeds resulting from such Transfer (or a related series of Transfers) is less than $20,000,000, or (B) the Transfer of First Savings Mortgage Loans to First Savings Mortgage Corporation pursuant to the First Savings Repurchase Agreement, and (iii) except for a Collateral Disposition of whole Mortgage Loans securing Eligible Warehouse Loans, the Collateral Disposition Proceeds deposited as provided in clause (i) above represents the fair market value of such the Assets Transferred.”

(k) Section 7.02(q) of the Loan Agreement is hereby amended and restated in its entirety to read as follows:

“(q) without the prior written consent of the Lender Agent, terminate the administrator under the European Security Documents or take any other action under the European Security Documents which could reasonably be expected to adversely affect the Lender or the Lender Agent or the value of the related European Note, provided , however, that this Section shall not apply to automatic termination of the administrator under the applicable agreements;”

(l) Section 12.11(b) if the Loan Agreement is hereby amended and restated to read as follows:

“(b) No Collateral shall be released from the security interest created by the Security Agreement without the prior written consent of the Lender Agent; provided that (a) if whole Mortgage Loans are Transferred in the ordinary course of an Obligor’s business and the Collateral Disposition Proceeds in respect of such Transfer (or a related series of Transfers) is less than $20,000,000, or (b) if First Savings Mortgage Loans are Transferred to First Savings Mortgage Corporation pursuant to the First Savings Repurchase Agreement, then the Lien on such Collateral shall be released automatically concurrently with such Transfer in accordance with the Security Agreement and any related Collateral Addition Designation Notice.”

(m) Section 13.05 of the Loan Agreement is hereby amended by amending and replacing the second sentence of Section 13.05 in its entirety to read as follows:

“EACH PARTY HERETO HEREBY IRREVOCABLY SUBMITS TO THE NONEXCLUSIVE JURISDICTION OF THE COURTS OF THE STATE OF NEW YORK LOCATED IN THE CITY OF MANHATTAN OR IN THE UNITED STATES DISTRICT COURT FOR THE SOUTHERN DISTRICT OF NEW YORK FOR PURPOSES OF ALL LEGAL PROCEEDINGS ARISING OUT OF OR RELATING TO THIS AGREEMENT OR THE TRANSACTIONS CONTEMPLATED HEREBY AND IRREVOCABLY AGREES TO BE BOUND BY ANY JUDGMENT RENDERED THEREBY IN CONNECTION WITH SUCH LITIGATION.”

SECTION 2.2 Amendments to Definitions . Each of the parties hereto hereby consents and agrees that Schedule 1.01 to the Loan Agreement shall be amended as of the Amendment Effective Date as follows:

(a) Each of the following definitions in Schedule 1.01 to the Loan Agreement is hereby amended and restated in full to read as follows:

“ Applicable Deposit Date ” means:

(i) in the case of any Collateral Disposition Proceeds or other Collections relating to a European Note:

(a) one (1) Business Day after such funds are received in relation to a European Note if such Collateral Disposition Proceeds or other Collections exceed $100,000,000;

(b) in all other cases relating to Collateral Disposition Proceeds, not later than five Business Days after the earlier of (I) the next monthly Distribution Date or Redemption Date (as defined in the Flume No. 8 Security Documents) with respect to Collateral Dispositions relating to the Flume No. 8 Note or the Note Payment Date (as defined in the GX II Security Documents) with respect to Collateral Dispositions relating to the GX II Note

or (II) the date the amount of Collateral Disposition Proceeds on deposit in (1) the applicable accounts relating to the English SPE exceeds £5,000,000 in the aggregate or (2) the applicable accounts relating to the GX II SPE exceeds EUR5,000,000 in the aggregate, as applicable; and

(c) in all other cases relating to Collections, not later than five Business Days after the next monthly Distribution Date (as defined in the Flume No. 8 Security Documents) with respect to Collections relating to the Flume No. 8 Note or the Note Payment Date (as defined in the GX II Security Documents) with respect to Collections (excluding Collateral Disposition Proceeds) relating to the GX II Note.”

(ii) in the case of Scheduled Collections on US Mortgage Loans, twenty calendar days after the month in which such Scheduled Collections were received by the related servicer; and

(iii) in the case of any Collateral Disposition Proceeds or other Collections relating to an Asset (other than as described in clause (i) or (ii) above), (a) the day such funds are received by an Obligor if such Collateral Disposition Proceeds or other Collections exceed $100,000,000 or (b) in all other cases, three Business Days after such funds are received by an Obligor; provided that, solely with respect to US Mortgage Loans, prior to June 18, 2009 the Obligors shall only be required to use commercially reasonable efforts to remit Collateral Disposition Proceeds in accordance with this clause (iii) .”

“ Approved Additional Collateral ” means (a) Eligible Assets consisting of US Mortgage Loans, (b) Eligible Assets consisting of Incremental Advances, (c) Eligible Assets consisting of an increase in the aggregate outstanding principal balance of a European Note in accordance with the terms of the European Security Documents, (d) Eligible Assets consisting of an increase in the Carrying Value of RFC’s equity interest in Equity Investment I due to its acquisition of Supporting Assets and (e) any other Eligible Assets which have been identified as Eligible Assets in a Collateral Addition Designation Notice.

“ Approved Exceptions ” means, with respect to an Asset, any irregularity in the documentation, underwriting or origination for such Asset, such irregularity (i) does not make the related Contracts unenforceable and is not reasonably expected to impair the practical realization of benefits intended to be created by such Contracts (determined without giving effect to any indemnification from the related Originator), (ii) has been taken into account by the Obligors in determining the Carrying Value of such Asset in accordance with its usual and customary business practices, and (iii) has been disclosed to the Lender Agent, it being understood and agreed that disclosure may be in the form of discussion between the

Obligors and the Lender Agent in the context of the Lender Agent’s determination of whether to include such Asset (individually or with a group of Assets) in a Collateral Addition Designation Notice and, if applicable, establishment of a related Specified Percentage (as defined in Schedule 2.04 ), provided that the Obligors’ description of such irregularity is sufficiently clear and detailed to enable the Lender to reasonably evaluate the risks with respect to such Asset.

“ Asset ” means (i) a Mortgage Loan, (ii) a Financial Asset Backed Security, (iii) an Eligible Warehouse Loan or a increase in the Collateral Value thereof, (iv) a Flume No. 8 Note or an increase in the Carrying Value thereof due to Flume No. 8 SPE’s acquisition of Supporting Assets, (v) the GSAP Class A-1 Preference Shares and the GSAP Class A-2 Preference Shares, (vi) an Equity Interest in a Financing SPV or an increase in the Carrying Value thereof due to its acquisition of Supporting Assets, (vii) a GX II Note or an increase in the Carrying Value thereof due to GX II SPE’s acquisition of Supporting Assets, (viii) RFC’s equity interest in Equity Investment I or an increase in the Carrying Value thereof due to its acquisition of Supporting Assets, or (ix) any other asset identified as an Asset in a Collateral Addition Designation Notice.

“ Collateral Addition Designation Notice ” means a notice in writing (which may be electronic) delivered by the Lender Agent at a Borrower’s request with respect to any category or sub-category of Approved Additional Collateral, which notice shall approve or designate a Collateral Addition Date for such Approved Additional Collateral as well as any applicable advance rates, additional eligibility requirements, opinion requirements, or other restrictions, terms or conditions as the Lender Agent may specify in its discretion; it being understood that this Agreement and the other Facility Documents may refer to a category of Collateral prior to the Collateral Addition Date therefor, but that such references will not be given effect until such Collateral Addition Date.

“ Collateral Value Report ” means a Collateral Value Report, substantially in the form of Exhibit 2.04(a) , with additional information on Equity Investment I, US Mortgage Loans and REO Property, substantially in the format provided pursuant to the Senior Debt Facility Documents, as the Lender Agent may reasonably request from time to time, delivered by the Borrowers to the Lender Agent in accordance with Sections 2.04(a)-(b) .”

“ Custodial Transfer Date ” means (i) May 19, 2009 or (ii) such other date as the Borrowers, the Guarantors and the Lender Agent shall agree in writing.

“ Eligible Asset ” means an Asset which satisfies the following criteria:

(a) such Asset (i) is an Incremental Advance, (ii) is an increase in the aggregate outstanding principal balance of a European Note in accordance with the terms of the applicable European Security Documents with the consent

of the Lender Agent in its sole discretion in an effective Collateral Addition Designation Notice, (iii) RFC’s equity interest in Equity Investment I and, with the consent of the Lender Agent in its sole discretion in an effective Collateral Addition Designation Notice, an increase in the Carrying Value of such equity interest due to its acquisition of Supporting Assets, (iv) is a US Mortgage Loan listed on a Mortgage Schedule in accordance with Section 7.01(w) and designated as an Eligible Asset in a Collateral Addition Designation Notice, or (v) is otherwise specified as an Eligible Asset with the consent of the Lender Agent in its sole discretion in an effective Collateral Addition Designation Notice;

(b) such Asset is owned by a Borrower or a Guarantor; and

(c) such Asset satisfies the Eligibility Requirements.

“ Eligible Warehouse Loans ” mean loans or other extensions of credit made by a Guarantor pursuant to an Eligible Warehouse Agreement to the borrower or counterparty under such agreement, provided that such loan satisfies the Eligibility Requirements; and provided further that, for the avoidance of doubt, the Eligibility Requirements shall not apply to any mortgage loans or other loans which secure an Eligible Warehouse Loan or are the subject of purchase pursuant to an Eligible Warehouse Agreement).

“ European Hedging Account ” means, (a) prior to the Account Addition Date, the Existing European Hedging Account and (b) after the Account Addition Date, one or more newly established segregated trust accounts established in the name of the Borrowers in the United Kingdom into which solely proceeds of the European Notes are deposited and each at all times after the Account Addition Date subject to an account control arrangement acceptable to the Lender Agent in its sole discretion.

“ European SPV Accounts ” means each account held in the name of the Flume No. 8 SPV or the GX II SPE pursuant to the terms of the Flume No. 8 Security Documents or the GX II Security Documents (as applicable).

“ First Savings Warehouse Agreement ” means, collectively, the First Savings Credit Agreement and the First Savings Repurchase Agreement.

“ GMAC Bank ” means, collectively, (i) IB Finance Holding Company, (ii) Ally Bank, formerly known as GMAC Bank and (iii) any successor thereto.

“ Mortgage Loan ” means any first or second lien residential mortgage loan.

“ Security Documents ” means (a) the Security Agreement, the Omnibus Security Agreement, the Account Control Agreements, the GSAP Indenture Transaction Documents, the Flume No. 8 Security Documents, the GX II Security Documents and the Custody Agreements, (b) any document designated as a Security Document in any Collateral Addition

Designation Notice and (c) all of the security agreements, pledges, collateral assignments, mortgages, deeds of trust, trust deeds or other instruments evidencing or creating or purporting to create any security interests in favor of the Omnibus Agent or Lender Agent for its benefit and for the benefit of the Lender Parties.

“ Underlying Documents ” means (a) (i) the First Savings Warehouse Agreement, the Provident Warehouse Agreement and the other Warehouse Facility Documents; (ii) the Flume No. 8 Note, the Flume No. 8 Security Documents and the other Flume No. 8 Facility Documents; (iii) the GSAP Class A-1 Preference Shares, the GSAP Class A-2 Preference Shares and the GSAP Indenture Transaction Documents and (iv) the GX II Note and the GX II Security Documents; and (b) after the applicable Collateral Addition Date, any document designated as an Underlying Document in any Collateral Addition Designation Notice.

(b) The definition of “ Financing SPV ” in Schedule 1.01 to the Loan Agreement is hereby amended by replacing “the Flume No. 8 SPE” with “the European SPEs”.

SECTION 2.3 Additional Definitions . Each of the parties hereto hereby consents and agrees that Schedule 1.01 to the Loan Agreement shall be amended as of the Amendment Effective Date by adding the following new definitions in the appropriate alphabetical location therein:

“ Collateral Disposition Proceeds ” means, with respect to an Asset, the net proceeds received by an Obligor from the Collateral Disposition of such Asset.

“ Custody Agreement ” has the meaning specified in Section 7.01(m) .

“ Cut-Off Date ” means, with respect to a Monthly Report, the last day of the calendar month immediately prior to the month in which such Monthly Report is required to be delivered.

“ Equity Investment I ” means Equity Investment I, LLC, a Delaware limited liability company.

“ European Note ” means a Flume No. 8 Note and/or GX II Note (as applicable).

“ European Security Documents ” means the Flume No. 8 Security Documents or the GX II Security Documents (as applicable).

“ First Savings Mortgage Loan ” means any mortgage or other loan subject to the First Savings Repurchase Agreement.

“ First Savings Credit Agreement ” means the Sixth Amended and Restated Warehousing Credit and Security Agreement dated as of April 1, 2005 between RFC and First Savings Mortgage Corporation, as the same may be amended, supplemented, restated or otherwise modified from time to time.

“ First Savings Repurchase Agreement ” means the master repurchase agreement entered into between RFC and First Savings Mortgage Corporation in May or June 200