EXCHANGE AGREEMENT

Exhibit 2.6

EXECUTION VERSION

EXCHANGE AGREEMENT

DATED AS OF SEPTEMBER 30, 2008,

AMONG

HANOVER CAPITAL MORTGAGE HOLDINGS, INC.,

AMSTER TRADING COMPANY

AND

RAMAT SECURITIES, LTD

TABLE OF CONTENTS

     EXCHANGE AGREEMENT, dated as of September 30, 2008 (this “ Agreement ”), among Hanover Capital Mortgage Holdings, Inc., a Maryland corporation (“ Hanover ”), Amster Trading Company, an Ohio corporation, and Ramat Securities, LTD, an Ohio limited liability company (each a “ Seller ” and, collectively, the “ Sellers ”).

W I T N E S S E T H :

     WHEREAS, the Sellers own all of the Preferred Securities (as defined in the Trust Agreement (defined below)), representing undivided beneficial interests in the assets of Hanover Statutory Trust II (the “ Trust ”), each having a Liquidation Amount of $1,000, and having an aggregate Liquidation Amount of $20,000,000, provided for in that certain Amended and Restated Declaration of Trust, dated as of November 4, 2005, among Hanover, Wilmington Trust Company, as Institutional trustee and Delaware trustee (the “ Institutional Trustee ”), the administrative trustees named therein (the “ Administrative Trustees ”) and the holders from time to time of the individual beneficial interests in the asset of the trust (the “ Trust Agreement ” and such beneficial interests in the Trust, the “ Preferred Securities ”). Capitalized terms used but not otherwise defined herein shall have the meanings ascribed to such terms in the Trust Agreement;

     WHEREAS, concurrently with the execution of this Agreement, Walter Industries, Inc. (“ Walter ”), JWH Holding Company, LLC (“ JWH ”), and Hanover are entering into an Agreement and Plan of Merger, dated as of the date hereof (as amended, supplemented, restated or otherwise modified from time to time, the “ Merger Agreement ”) pursuant to which, among other things, JWH will merge into Hanover (the “ Merger ”), the separate existence of JWH shall cease and Hanover shall continue as the surviving corporation and, except as otherwise provided in the Merger Agreement, shares of common stock of Hanover, par value $0.01 per share (the “ Common Stock ”), issued and outstanding immediately prior to the date and time at which the Merger shall become effective (the “ Merger Effective Time ”) shall be combined into fully paid and non-assessable shares of common stock of the surviving corporation (“ Surviving Corporation Common Stock ”) at the rate specified in the Merger Agreement;

     WHEREAS, in connection with the transactions contemplated by this Agreement, the Sellers, Hanover and the other signatories thereto have entered into and executed the Voting Agreement (as defined in Article I), pursuant to which each of the parties thereto other than Hanover have agreed to, among other things, vote any shares of Common Stock it now holds or will in the future hold in favor of the Merger; and

     WHEREAS, the parties hereto desire to make certain representations, warranties, covenants and agreements in connection with the Exchange (as defined in Section 2.1) and also to prescribe certain conditions to the Exchange.

     NOW, THEREFORE, in consideration of the representations, warranties, covenants, agreements and conditions hereafter set forth, and intending to be legally bound hereby, the parties hereto agree as follows:

ARTICLE I

DEFINED TERMS

     As used in this Agreement, the following terms have the meanings ascribed thereto:

     “Affiliate” shall mean, with respect to any specified Person, any other Person that, directly or indirectly, controls, is controlled by or is under common control with, such specified Person. For purposes of this definition, “control” (including, with correlative meanings, the terms “controlled by” and “under common control with”), as used with respect to any Person, shall mean the possession, directly or indirectly, of the power to direct or cause the direction of the management or policies of such Person, whether through the ownership of voting securities, by Contract or otherwise.

     “Agreement” shall mean this Agreement, together with all exhibits attached hereto and the Disclosure Letters.

     “Common Stock Consideration” shall mean 6,762,793 duly authorized, validly issued, fully paid and non-assessable shares of Common Stock.

     “Confidentiality Agreement” shall have the meaning ascribed to it in Section 5.1.

     “Contract” shall mean any written loan or credit agreement, note, bond, debenture, indenture, mortgage, guarantee, deed of trust, lease, franchise, permit, authorization, license, contract, instrument, employee benefit plan or practice or other binding agreement, obligation, arrangement, understanding or commitment.

     “Exchange Act” shall mean the Securities Exchange Act of 1934, as amended, together with the rules and regulations of the SEC promulgated thereunder.

     “Governmental Authority” shall mean any nation or government or any agency, public of regulatory authority, instrumentality, department, commission, court, arbitrator, ministry, tribunal or board of any nation or government or political subdivision thereof, in each case, whether foreign or domestic and whether national, supranational, federal, tribal, provincial, state, regional, local or municipal.

     “Hanover SEC Documents” shall mean all forms, reports, statements, certifications and other documents (including all exhibits, amendments and supplements thereto) required to be filed by Hanover with the SEC since January 1, 2007.

     “Hanover Stock Plans” shall mean the Hanover 1999 Equity Incentive Plan and the Hanover 1997 Executive and Non-Employee Director Stock Option Plan.

     “Law” means applicable statutes, common laws, rules, regulations, codes, licensing requirements, judgments, injunctions, writs, decrees, licenses, ordinances, authorizations, permits, certificates, easements, variances, exemptions, consents, orders,

franchises, approvals, governmental guidelines, standards or interpretations having the force of law, rules and bylaws, in each case, of or administered by a Governmental Authority.

     “Lien” shall mean, with respect to any property or asset, any mortgage, easement, lien, pledge (including any negative pledge), charge, option, right of first or last refusal or offer, security interest or encumbrance of any kind in respect of such property or asset.

     “Person” shall mean a natural person, corporation, limited liability company, partnership, limited partnership or other entity, including a Governmental Authority.

     “Proxy Statement/Prospectus” shall mean the proxy statement/prospectus to be distributed to the Hanover stockholders and the Walter stockholders in connection with the Merger and the transactions contemplated by the Merger Agreement, including any preliminary proxy statement/prospectus or definitive proxy statement/prospectus filed with the SEC in accordance with the terms and provisions thereof and prepared in accordance with applicable Law. The Proxy Statement/Prospectus shall constitute a part of the Registration Statement.

     “Registration Statement” shall mean the Registration Statement on Form S-4 to be filed by Hanover with the SEC to effect the registration under the Securities Act of the shares of Hanover Common Stock to be issued to holders of limited liability company interests of JWH pursuant to the Merger and prepared in accordance with applicable Law.

     “Requisite Approval” shall have the meaning ascribed to it in Section 3.4.

     “SEC” shall mean the U.S. Securities and Exchange Commission.

     “Securities Act” shall mean the Securities Act of 1933, as amended, together with the rules and regulations of the SEC promulgated thereunder.

     “Subsidiaries” shall mean, with respect to any Person, another Person (i) of which 50% or more of the capital stock, voting securities, other voting ownership or voting partnership interests having voting power under ordinary circumstances to elect directors or similar members of the governing body of such corporation or other entity (or, if there are no such voting interests, 50% or more of the equity interests) are owned or controlled, directly or indirectly, by such first Person or (ii) of which such first Person is a general partner.

     “Transaction Agreements” shall mean (i) the Merger Agreement, (ii) the Exchange Agreement by and between Hanover and Taberna Preferred Funding I, Ltd. (“Taberna”) dated as of the date hereof (the “Taberna Agreement”), (iii) the Voting Agreement, and (iv) the Confidentiality Agreement.

     “Voting Agreement” shall mean the Voting Agreement, dated as of the date hereof, among Walter, JWH, Sellers and certain other stockholders of Hanover.

ARTICLE II

THE TRANSACTION

     Section 2.1 The Exchange . Subject to the terms and conditions of this Agreement, immediately prior to the Merger Effective Time, each of the Sellers hereby agrees to sell, transfer, assign, convey and deliver to Hanover, and Hanover hereby agrees to acquire from the Sellers, all of the Preferred Securities owned by the Sellers free and clear of all Liens (the “ Sellers’ Preferred Securities ”) in exchange for (i) the Common Stock Consideration and (ii) an aggregate cash payment by Hanover of $750,000 (the “ Exchange ”).

     Section 2.2 Closing . A closing of the Exchange (the “ Closing ”) shall be held substantially concurrently with the closing of the Merger (the “ Closing Date ”); provided that all conditions set forth in Article VI (other than those conditions that by their nature are to be satisfied or waived at the Closing, but subject to the satisfaction or waiver of those conditions) are satisfied or waived.

     Section 2.3 Delivery by Sellers of Preferred Securities . At the Closing, the Sellers shall: (i) surrender for transfer the Preferred Securities Certificate(s) representing each of the Seller’s Preferred Securities duly endorsed and accompanied by a written instrument of transfer in form satisfactory to the Trust, the Registrar and to Hanover, duly executed by the applicable Seller and accompanied (to the extent required by the Trust or the Registrar) by a certificate of Hanover substantially in the form of Exhibit B or C to the Trust Agreement; (ii) take (or shall have taken, to the extent prior action is required) (A) all other actions required pursuant to the Trust Agreement or as reasonably requested by Hanover, the Registrar, the Trust, the Administrative Trustees or the Institutional Trustee to cause the Trust to issue and the Institutional Trustee to authenticate and deliver, in the name of Hanover, one or more new Preferred Securities Certificates in authorized denominations of an aggregate Liquidation Amount equal to that of each of the Sellers’ Preferred Securities, as may be required by the Trust Agreement, dated the Closing Date, all as described in Article VIII of the Trust Agreement and (B) at the request of Hanover, all such reasonable actions as may be necessary and appropriate to cause the cancellation of the Preferred Securities; and (iii) take all other actions as may be necessary and appropriate to vest in Hanover good and marketable title to the Preferred Securities free and clear of any and all Liens.

     Section 2.4 Issuance of Common Stock Consideration and Payment of Cash . At the Closing, (i) Hanover will deliver to the Sellers stock certificates representing the Common Stock Consideration, and (ii) Hanover will make payment to the Sellers of $750,000 (together with the Common Stock Consideration, the “ Consideration ”) by wire transfer of immediately available funds to an account designated by the Sellers in writing no later than five business days prior to the date of the meeting of Hanover stockholders called to approve the Merger. The Common Stock Consideration and the cash Consideration shall be allocated between the Sellers on a pro rata basis.

     Section 2.5 Release . In exchange for the valuable consideration set forth above and other valuable consideration, the receipt and adequacy of which are herein acknowledged, and effective as of the date hereof, provided that this Section 2.5 shall cease to be

effective and shall be null and void ab initio if and when the Merger Agreement is terminated in accordance with its terms:

          (a) Hanover and its officers, directors, employees, attorneys, agents, heirs, executors, administrators, parents, subsidiaries, affiliates, successors and assigns (collectively “ Hanover Releasor ”) unconditionally and irrevocably forever release and discharge the Sellers and their respective officers, directors, employees, attorneys, agents, representatives, heirs, executors, administrators, direct and indirect parents, subsidiaries, affiliates, predecessors, successors and assigns (collectively “ Seller Releasee ”) from all actions, causes of action, suits, debts, dues, sums of money, accounts, reckonings, bonds, bills, specialties, covenants, contracts, controversies, agreements, promises, variances, trespasses, damages, judgments, extents, executions, claims, and demands whatsoever, in law or equity, whether known or unknown (“ Claims ”), which against the Seller Releasee the Hanover Releasor ever had, now has or hereafter can, shall or may have from the beginning of time arising out of or in connection with the Trust Agreement, Indenture or either of the Sellers’ Preferred Securities, whether arising in equity or pursuant to any law, rule or regulation, including any Claims of which Hanover Releasor is not aware or does not suspect to exist as of the date on which Hanover Releasor signs this Agreement; and

          (b) Each Seller and its respective officers, directors, employees, attorneys, agents, heirs, executors, administrators, parents, subsidiaries, affiliates, successors and assigns (collectively “ Seller Releasor ”) unconditionally and irrevocably forever release and discharge Hanover and its respective officers, directors, employees, attorneys, agents, representatives, heirs, executors, administrators, direct and indirect parents, subsidiaries, affiliates, predecessors, successors and assigns (collectively “ Hanover Releasee ” and together with the Seller Releasee, the “ Released Parties ”) from all Claims, which against the Hanover Releasee the Seller Releasor ever had, now has or hereafter can, shall or may have from the beginning of time arising out of or in connection with the Trust Agreement, Indenture or either of the Sellers’ Preferred Securities, whether arising in equity or pursuant to any law, rule or regulation, including any Claims of which Seller Releasor is not aware or does not suspect to exist as of the date on which Seller Releasor signs this Agreement.

ARTICLE III

REPRESENTATIONS AND WARRANTIES OF HANOVER

     Except as disclosed in the Hanover SEC Documents (other than any disclosures included in such filings that are predictive, speculative or forward-looking in nature, including any disclosures in any “Risk Factors” sections thereof) or as expressly contemplated by the Transaction Agreements, Hanover represents and warrants to the Sellers as follows:

     Section 3.1 Organization, Qualification, Etc . Hanover is a corporation duly organized, validly existing and in good standing under the Laws of the State of Maryland. Hanover has all requisite power and authority to own or lease and operate and use its properties and assets and carry on its business as presently conducted and is duly qualified and licensed to do business and is in good standing in each jurisdiction in which the ownership or leasing of its property or the conduct of its business requires such qualification, except for jurisdictions in

which the failure to be so qualified or to be in good standing would not, individually or in the aggregate, reasonably be expected to materially impair or delay Hanover’s ability to perform each of its obligations hereunder and to consummate the transactions contemplated hereby. Each of the Hanover Subsidiaries is a corporation or other legal entity duly organized, validly existing and, to the extent such concept or similar concept exists in the relevant jurisdiction, in good standing under the laws of the state or other jurisdiction of its incorporation or other organization, has all requisite power and authority to own or lease and operate and use its properties and assets and to carry on its business as presently conducted and is duly qualified and licensed to do business and is in good standing in each jurisdiction in which the ownership or leasing of its property or the conduct of its business requires such qualification, except for jurisdictions in which the failure to be so qualified or to be in good standing would not, individually or in the aggregate, reasonably be expected to materially impair or delay Hanover’s ability to perform each of its obligations hereunder and to consummate the transactions contemplated hereby.

     Section 3.2 Corporate Authority; No Violation, Etc . Hanover has the requisite corporate power and authority to enter into this Agreement and each agreement or instrument to be executed and delivered in connection with or pursuant hereto and, subject in the case of this Agreement to obtaining the Requisite Approval, to perform its obligations hereunder and to consummate the transactions contemplated hereby. The execution, delivery and performance by Hanover of this Agreement and the consummation of the transactions contemplated hereby have been duly authorized by all requisite corporate action on the part of Hanover, subject, in the case of the performance of this Agreement and the consummation of the transactions contemplated hereby, to obtaining the Requisite Approval. This Agreement has been duly executed and delivered by Hanover and, assuming due authorization, execution and delivery of this Agreement by each Seller, constitutes a legal, valid and binding agreement of Hanover, enforceable against Hanover in accordance with its terms, subject to bankruptcy, insolvency, reorganization, moratorium and similar laws of general application relating to or affecting creditors’ rights and to general equity principles. Neither the execution and delivery by Hanover of this Agreement, the consummation by Hanover of the transactions contemplated hereby nor compliance by Hanover with any of the provisions hereof (i) violates or conflicts with any provisions of Hanover’s charter or bylaws, (ii) requires any consent, approval, authorization or permit of, registration, declaration or filing with, or notification to, any Governmental Authority or any other Person, other than the Requisite Approval, (iii) results in a default (or an event that, with notice or lapse of time or both, would become a default) or gives rise to any right of termination or buy-out by any third party, cancellation, amendment or acceleration of any obligation or the loss of any benefit under any Contract to which Hanover or any of its Subsidiaries is a party or by which Hanover or any of its Subsidiaries or any of their respective assets or properties is bound or affected, (iv) results in the creation of a Lien on any of the issued and outstanding shares of Common Stock or equity securities of any Subsidiary or on any of the assets of Hanover or its Subsidiaries or (v) violates or conflicts with any Law applicable to Hanover or any of its Subsidiaries, or any of the properties, businesses or assets of any of the foregoing, other than such exceptions in the case of each of clauses (ii), (iii), (iv