INTERCREDITOR AGREEMENT

Exhibit 4.21

EXECUTION COPY

COLLATERAL AGENCY AND

INTERCREDITOR AGREEMENT

Collateral Agency and Intercreditor Agreement, dated as of May 10, 2005 (this “Agreement”), is by and among CALYON NEW YORK BRANCH (“Calyon”), and LEHMAN BROTHERS SPECIAL FINANCING INC., in its individual capacity (“Lehman” and, together with Calyon, the “Swap Counterparties”) and as agent under this Agreement for Lehman and Calyon (in such capacity and together with any successor thereto in such capacity, “Collateral Agent”).

Recitals:

WHEREAS, Lehman has entered into that certain ISDA Master Agreement, dated as of May 10, 2005 (including the Schedule, each Confirmation and Credit Support thereto, the “Lehman Swap”), between Lehman and Accredited Home Lenders, Inc. (“Accredited”);

WHEREAS, Calyon has entered into that certain ISDA Master Agreement, dated as of May 10, 2005 (including the Schedule and each Confirmation and Credit Support Annex thereto, the “Calyon Swap” and, together with the Lehman Swap, the “Swaps”), between Calyon and Accredited;

WHEREAS, Accredited has pledged pursuant to the Credit Support Annex to the Lehman Swap (the “Lehman CSA”) certain posted debt obligations, securities and cash and its hedges and other assets (collectively, the “Lehman Collateral”) to support its obligations under the Lehman Swap (the “Lehman Obligations”);

WHEREAS, Accredited has pledged pursuant to the Credit Support Annex to the Calyon Swap (the “Calyon CSA” and, together with the Lehman CSA, the “Security Documents”) certain posted debt obligations, securities and cash and its hedges and other assets (the “Calyon Collateral” and, together with the Lehman Collateral, the “Collateral”) to support its obligations under the Calyon Swap (the “Calyon Obligations” and, together with the Lehman Obligations, the “Obligations”);

WHEREAS, the Swap Counterparties desire to appoint Lehman as Collateral Agent under this Agreement and the Security Documents and to define the rights and duties of the Collateral Agent and the Collateral Agent desires to set forth the terms and conditions upon which it shall accept such agency; and

WHEREAS, the Swap Counterparties also desire to enter into agreements with one another as to certain matters relating to the Security Documents, including the administration of liens and the application of the proceeds of the Collateral.

NOW THEREFORE, in consideration of the foregoing and the mutual covenants herein contained, and for other good and valuable consideration, the receipt and sufficiency of which is hereby acknowledged, the parties hereto agree as follows:

1. Enforcement of Collateral.

1.1 Events of Enforcement. Anything contained in the Security Documents to the contrary notwithstanding, the Swap Counterparties agree among themselves and for their own benefit alone that, except as expressly provided otherwise in this Agreement (including, without limitation, actions permitted pursuant to the provisions of Section 4.4), the Collateral Agent shall only be authorized to take such actions under the Security Documents and to enforce or prepare to enforce such remedies available under such Security Documents as it may be directed to take or enforce pursuant to the written instruction of the Instructing Swap Counterparty (as defined in Section 4.7) (each such instruction being herein called an “Event of Enforcement”). In furtherance of the foregoing, the Collateral Agent agrees to make such demands and give such notice under the Security Documents as may be requested by, and to take such action to enforce the Security Documents and to foreclose upon, collect and dispose of the Collateral or any portion thereof as may be directed by, the Instructing Swap Counterparty; provided, however, that (i) the Collateral Agent shall not be required to take any action that it has been advised by legal counsel is contrary to law or the terms of the Swaps, the Security Documents, or this Agreement and (ii) the Collateral Agent shall not be required to take any action unless indemnified in accordance with the provisions of Section 4.6 hereof.

1.2 Collateral Agent’s Enforcement Against Collateral. Upon the occurrence of any Event of Enforcement, the Collateral Agent shall, on behalf of the Swap Counterparties, seek to realize upon the liens and security interests and other rights granted and provided for in the Security Documents in accordance with the instructions of the Instructing Swap Counterparty, but in the absence of such instructions, then in such manner as it deems appropriate and in that regard shall have the right to incur costs and expenses (including reasonable costs and expenses of attorneys) which, to the extent are not paid by Accredited or out of the proceeds of the Collateral, shall be shared by the Swap Counterparties in the manner provided for in Section 4.11 of this Agreement. The Collateral Agent shall not be obligated to exhaust its remedies against the Collateral or Accredited prior to seeking reimbursement for its costs and expenses from the Swap Counterparties.

1.3 Restrictions on Actions. Each Swap Counterparty agrees that, so long as any Obligations are outstanding, the provisions of this Agreement shall provide the exclusive method by which any Swap Counterparty may realize the benefits afforded by the Security Documents. Each Swap Counterparty shall, for the mutual benefit of all Swap Counterparties, except as expressly provided otherwise by this Agreement:

(a) refrain from taking or filing any action, judicial or otherwise, to enforce rights or pursue any remedy with respect to or under the Security Documents, except for delivering notices hereunder;

2

(b) refrain from (i) selling any Obligations to Accredited or any affiliate of Accredited, and (ii) accepting any guaranty of, or any other security for, the Obligations from Accredited or any affiliate of Accredited, except any guaranty or security granted to the Collateral Agent for the benefit of all Swap Counterparties; and

(c) refrain (except in connection with action taken by the Instructing Swap Counterparty) from exercising or requesting the Collateral Agent to exercise any rights or remedies under the Security Documents which have or may have arisen or which may arise as a result of an event of default or event of termination (as such terms are defined in the Swaps, an “Event of Default” or “Termination Event”) under any Swap;

provided, however, that nothing contained in subsections (a) through (c) above shall prevent any Swap Counterparty from imposing a default interest in accordance with the Swaps, or prevent a Swap Counterparty from raising any defenses in any action in which it has been made a party defendant or has been joined as a third party; provided, further that nothing contained herein shall prevent any Swap Counterparty from accepting a guaranty from any person or entity which, contemporaneously with the granting of such guaranty to such Swap Counterparty, is granting a guaranty or guaranties to the other Swap Counterparty to secure the Obligations owed to them, which guaranty or guarantees has or have the same or substantially the same terms as the terms of the guaranty or guarantees given to such Swap Counterparty; and provided, further that nothing contained herein shall affect or impair the right any Swap Counterparty may have under the terms and conditions governing the Obligations owing to such Swap Counterparty to accelerate, demand and take any action to enforce repayment of such Obligations or to file a lawsuit and obtain and enforce a judgment against Accredited, subject to the provisions of Article 3 hereof.

1.4 Terms of Obligations. Subject to the terms of the Swaps, the terms and conditions applicable to the Obligations owing to each Swap Counterparty by Accredited shall be such as such Swap Counterparty and Accredited shall from time to time agree upon and may be changed or modified at any time in accordance with the applicable Swap without in any manner affecting this Agreement or impairing this Agreement.

2. Release of Collateral.

2.1 The Collateral Agent may, without notice to or the consent of any Swap Counterparty, release from the liens and security interests of the Security Documents any portion of the Collateral in connection with the sale, transfer or disposition of such Collateral by Accredited in the ordinary course of its business to the extent permitted by, and subject to the terms and conditions of, the Swaps.

3. Allocation of Proceeds of Collateral.

3.1 Application of Proceeds. The proceeds of the Collateral, including proceeds received through enforcement of the liens and security interests and other rights granted and provided for in the Security Documents, shall be applied and shared among the Swap Counterparties as follows:

(a) First, to the payment of any outstanding costs and expenses incurred by the Collateral Agent in enforcing such liens and security interests and in protecting and maintaining the Collateral and collecting the proceeds of the Collateral, to the extent that it has not theretofore been reimbursed for same by Accredited or out of the proceeds of the Collateral or, if not reimbursed promptly by Accredited or out of such proceeds, by the Swap Counterparties; and

3

(b) Second, to the Swap Counterparties ratably in accordance with the respective outstanding amount of Obligations owing to such Swap Counterparty under the Swaps for application to Obligations arising under and in connection with the Swaps.

3.2 Sharing of Proceeds.

(a) Subject to the remaining provisions of this Section 3.2, if a Swap Counterparty obtains any payments (other than scheduled payments on any Obligations, in each case received in the ordinary course either (i) prior to such Swap Counterparty’s receipt of written notice from another Swap Counterparty that an Event of Default or Termination Event has occurred and is continuing under which Accredited is the Defaulting Party or the Affected Party (as such terms are used in the Swaps) or (ii) after such Event of Default no longer shall be continuing) or proceeds of the Collateral, from Accredited with respect to any of the Obligations, including from the exercise of any setoff rights (hereinafter called a “Sharing Payment”), such Swap Counterparty shall either (i) promptly cause such amounts to be delivered to or put in the custody, possession or control of the Collateral Agent for disposition and distribution in accordance with Section 3.1 hereof or (ii) if such Swap Counterparty has not promptly complied with subclause (i) next above, promptly purchase from the remaining Swap Counterparties participations in the Obligations owing to the remaining Swap Counterparties and shall make such other adjustments from time to time as shall be equitable (herein, the purchase of such participations or the making of such other adjustments being called “Sharing Adjustments”) to the end that all Swap Counterparties shall share the benefit of such Sharing Payment pro rata in accordance with the respective outstanding principal amount of Obligations then owing to each Swap Counterparty.

(b) If, during the course of, or pursuant to, any bankruptcy, insolvency, reorganization, receivership, dissolution or similar proceeding or the assignment for the benefit of creditors or any other marshalling of assets (a “Bankruptcy Proceeding”) of Accredited, a Swap Counterparty (the “Returning Swap Counterparty”) is required by a court or other tribunal of competent jurisdiction, pursuant to Section 547 of the Bankruptcy Code or an analogous provision of other applicable law, to disgorge, refund, rebate or otherwise return any payment received for which there has been a distribution under Section 3.1 hereof to such Returning Swap Counterparty with respect to the Obligations (a “Disputed Payment”) to any trustee presiding over such Bankruptcy Proceeding or to any other person or entity, the other Swap Counterparty shall promptly pay to the Returning Swap Counterparty its respective pro rata shares of such Disputed Payment, such pro rata share being determined by multiplying the amount of the

4

Disputed Payment by a fraction, the numerator of which is the portion of such Disputed Payment received by such Swap Counterparty and the denominator of which is the amount of such Disputed Payment.

4. The Collateral Agent.

4.1 Appointment. Each of the Swap Counterparties hereby appoints Lehman as the initial Collateral Agent for the Swap Counterparties under, and subject to the provisions of, this Agreement, the Swaps and the Security Documents, and each of the Swap Counterparties authorizes the Collateral Agent to act as the agent of the Swap Counterparties under, and to effectuate the purposes of, this Agreement, the Swaps (to the extent of the duties delegated to the Collateral Agent thereunder) and the Security Documents (to the extent of the duties delegated to the Collateral Agent thereunder). The Collateral Agent agrees to act as such upon the express conditions contained in this Agreement.

4.2 Duties. (a) By the execution of this Agreement, each Swap Counterparty (i) authorizes the Collateral Agent to file all financing statements and amendments to and/or assignments of financing statements necessary to perfect the security interests granted pursuant to the Security Documents or to continue the perfection of such security interests as are covered by the financing statements in favor of Collateral Agent, and (ii) ratifies the Collateral Agent’s prefiling, if any, prior to the date hereof of financing statements as part of such perfection process. All of such financing statements or amendments to and/or assignments of financing statements shall name the Collateral Agent, as agent, as secured party for the benefit of all of the Swap Counterparties in accordance with the interests of each Swap Counterparty as described in this Agreement.

(b) Subject to the terms of this Agreement, the Swaps and the Security Documents, the Collateral Agent agrees to receive, hold, administer and enforce the Collateral and the Security Documents (including, without limitation, acting as Valuation Agent and making the calculation of Exposure (as defined in the Swaps), the determination of Value, Current Market Value and Exposure (as defined in the Swaps) and making market calls under the Security Documents), and to foreclose upon, collect and dispose of the Collateral and to apply the proceeds therefrom, in such manner and on such terms as are set forth herein and therein, and for the ratable benefit of the Swap Counterparties as provided herein and therein, and otherwise to perform its duties and obligations as Collateral Agent hereunder and under each Security Document to which it is a party in accordance with the respective terms hereof and thereof. Notwithstanding any provision to the contrary set forth elsewhere in this Agreement or the Security Documents, the Collateral Agent shall not have any duties or responsibilities in its capacity as Collateral Agent except those expressly set forth herein (including, without limitation, Section 4.4 hereof) or therein or any fiduciary relationship with any Swap Counterparty, and no implied duties shall be read into this Agreement, any Swap or any Security Document or otherwise exist with respect to the Collateral Agent, and the Collateral Agent and its directors, officers, employees, agents and representatives shall not be liable to any Swap Counterparty for any action taken or omitted to be taken by it or them under this Agreement, any

5