AMENDMENT NO. 13 TO LOAN AND SECURITY AGREEMENT

Exhibit 10.1

[Execution]

AMENDMENT NO. 13 TO LOAN AND SECURITY AGREEMENT

AMENDMENT NO. 13 TO LOAN AND SECURITY AGREEMENT, dated as of October 30, 2008 (this “Amendment No. 13”), entered into by and among Wachovia Bank, National Association, successor by merger to Congress Financial Corporation (Florida), in its capacity as agent acting for and on behalf of the parties to the Loan Agreement (as hereinafter defined) as lenders (in such capacity, “Agent”), the parties to the Loan Agreement as lenders (individually a “Lender” and collectively, “Lenders”), Supreme International, LLC, a Delaware limited liability company formerly known as Supreme International, Inc. (“Supreme”), Jantzen, LLC, a Delaware limited liability company formerly known as Jantzen, Inc. (“Jantzen”), Perry Ellis Menswear, LLC, a Delaware limited liability company formerly known as Perry Ellis Menswear, Inc. (“Perry Ellis Menswear”), Perry Ellis Europe Limited, a private limited company incorporated in England and Wales formerly known as Farah Manufacturing (U.K.) Limited (“Perry Europe”), Salant Holding, LLC, a Delaware limited liability company formerly known as Salant Holding Corporation (“Salant Holding” and together with Supreme, Jantzen, Perry Europe and Perry Ellis Menswear, each individually “Borrower” and collectively, “Borrowers”), Perry Ellis International, Inc., a Florida corporation (“Parent”), PEI Licensing, Inc., a Delaware corporation (“PEI Licensing”), Jantzen Apparel, LLC, a Delaware limited liability company formerly known as Jantzen Apparel Corp. (“Jantzen Apparel”), Supreme Real Estate I, LLC, a Florida limited liability company (“Supreme I”), Supreme Real Estate II, LLC, a Florida limited liability company (“Supreme II”), Supreme Realty, LLC, a Florida limited liability company (“Supreme Realty”), Supreme Munsingwear Canada Inc., a Canada corporation (“Supreme Canada”), Perry Ellis Shared Services Corporation, a Delaware corporation (“PE Shared Services”), Winnsboro DC, LLC, a Delaware limited liability company (“Winnsboro”), Tampa DC, LLC, a Delaware limited liability company (“Tampa DC”), Perry Ellis International Group Holdings Limited, a private company incorporated under the laws of Ireland having its principal place of business in the Bahamas (“Group Holdings”) and Perry Ellis Real Estate, LLC, a Delaware limited liability company formerly known as Perry Ellis Real Estate Corporation (“PE Real Estate” and, together, with Parent, PEI Licensing, Jantzen Apparel, Supreme I, Supreme II, Supreme Realty, Group Holdings, PE Shared Services, Winnsboro, Tampa DC, and Supreme Canada, each individually a “Guarantor” and collectively, “Guarantors”).

W I T N E S S E T H :

WHEREAS, Agent, Lenders, Borrowers and Guarantors have entered into financing arrangements pursuant to which Lenders (or Agent on behalf of Lenders) have made and may make loans and advances and provide other financial accommodations to Borrowers as set forth in the Loan and Security Agreement, dated October 1, 2002, by and among Agent, Lenders, Borrowers and Guarantors, as amended by Amendment No. 1 to Loan and Security Agreement, dated June 19, 2003, Amendment No. 2 to Loan and Security Agreement, dated September 22, 2003, Amendment No. 3 to Loan and Security Agreement, dated December 1, 2003, Amendment No. 4 to Loan and Security Agreement, dated February 25, 2004, Amendment No. 5 to Loan and Security Agreement,

dated July 1, 2004, Amendment No. 6 to Loan and Security Agreement, dated as of September 30, 2004, Amendment No. 7 to Loan and Security Agreement, dated as of February 26, 2005, Amendment No. 8 to Loan and Security Agreement, dated as of September 30, 2005, Amendment No. 9 to Loan and Security Agreement, dated as of February 24, 2006, Amendment No. 10 to Loan and Security Agreement, dated as of August 28, 2006, Amendment No 11 to Loan and Security Agreement, dated as of November 29, 2006 and Amendment No. 12 and Consent to Loan and Security Agreement, dated as of December 6, 2006 (as the same may hereafter be further amended, modified, supplemented, extended, renewed, restated or replaced, the “Loan Agreement”, and together with all agreements, documents and instruments at any time executed and/or delivered in connection therewith or related thereto, as from time to time amended, modified, supplemented, extended, renewed, restated, or replaced, collectively, the “Financing Agreements”);

WHEREAS, Borrowers and Guarantors have requested that Agent and Lenders agree to make certain amendments to the Loan Agreement, and Agent and Lenders are willing to agree to such amendments, subject to the terms and conditions set forth in this Amendment No. 13; and

WHEREAS, by this Amendment No. 13, Agent, Lenders, Borrowers and Guarantors desire and intend to evidence such amendments;

NOW, THEREFORE, in consideration of the foregoing, the mutual agreements and covenants contained herein, and other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the parties hereto agree as follows:

1. Definitions .

1.1 Additional Definitions .

(a) “Amendment No. 13” shall mean Amendment No. 13 to Loan and Security Agreement by and among Agent, Lenders, Borrowers and Guarantors, as the same now exists or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced.

(b) “Amendment No. 13 Effective Date” shall mean the first date on which all of the conditions precedent to the effectiveness of Amendment No. 13 shall have been satisfied and/or waived.

(c) “Incremental Facility” shall have the definition set forth in Section 2.4 of the Loan Agreement.

(d) “Maximum Credit Increase Effective Date” shall have the definition set forth in Section 2.4 of the Loan Agreement.

1.2 Amendments to Definitions .

(a) Each reference to the term “Applicable Margin” in the Loan Agreement or any other Financing Agreement is hereby amended to mean, at any time, as to the Interest Rate for Prime Rate Loans and the Interest Rate for Eurodollar Rate Loans

2

the applicable percentage (on a per annum basis) set forth below if the sum of (A) the Quarterly Average Excess Availability for the immediately preceding fiscal quarter plus (B) the Excess Cash as of the last day of such immediately preceding fiscal quarter, is at or within the amounts indicated for such percentage:

provided , that , (i) the Applicable Margin shall be calculated and established once each fiscal quarter (commencing with the fiscal quarter beginning November 1, 2008) and shall remain in effect until adjusted thereafter after the end of the next fiscal quarter and (ii) the Applicable Margin from the Amendment No. 13 Effective date through and including October 31, 2008, shall be the amount for Tier 2 set forth above.

(b) The definition of “Commitment” in Section 1.19 of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“Commitment” shall mean, at any time, as to each Lender, the principal amount set forth below such Lender’s signature on the signatures pages hereto designated as the Commitment or on Schedule 1 to the Assignment and Acceptance Agreement pursuant to which such Lender became a Lender hereunder in accordance with the provisions of Section 13.7 hereof, as the same may be adjusted from time to time in accordance with the terms hereof; sometimes being collectively referred to herein as “Commitments”.

(c) The definition of “Eligible Factor Receivables” in Section 1.29 of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“Eligible Factor Receivables” shall mean, as to each Borrower, Factor Receivables arising from the sale of Accounts by such Borrower to a Factor pursuant to and in accordance with the terms and conditions of the Factoring Agreements of such Borrower with such Factor, which are and continue to be acceptable to Agent based on the criteria set forth below,

3

less any charges, commissions, fees, chargebacks, deductions, setoffs and other amounts payable to such Factor; provided, that, regardless of the aggregate amount of Factor Receivables or Receivables that might satisfy the criteria so as to constitute Eligible Factor Receivables, in no event shall the aggregate amount of the Factor Receivables that constitute Eligible Factor Receivables be deemed to be more than $10,000,000 at any time. In general, subject to such limitation, Factor Receivables shall be Eligible Factor Receivables of a Borrower if: (i) the Accounts of such Borrower so sold to the Factor giving rise to such Factor Receivable have been approved by such Factor for its own credit risk and the sale of goods giving rise to such Accounts so sold have been approved and accepted by such Factor in accordance with the terms and conditions of the applicable Factoring Agreements (and such Accounts do not constitute “client risk” Accounts under the terms of the arrangements of such Borrower with such Factor); (ii) such Factor Receivable has been validly assigned by such Borrower to Agent and is payable to Agent pursuant to the Factor Assignment Agreement with the Factor obligated to pay such Factor Receivable; (iii) Agent shall have received a Factor Assignment Agreement duly authorized, executed and delivered by the Factor obligated in respect of such Factor Receivable and such Factor Assignment Agreement shall be in full force and effect and such Factor shall be in compliance in all respects with the terms thereof; (iv) such Factor Receivable is not unpaid after the date specified for payment under the terms of the Factoring Agreements applicable thereto; (v) such Factor Receivable is subject to the first priority, valid and perfected security interest of Agent and is not subject to any other security interest, pledge, lien, claim or other encumbrance except those permitted under this Agreement that are subject and subordinate to the security interests of Agent pursuant to an intercreditor agreement in form and substance satisfactory to Agent between the holder of such security interest or lien and Agent; (vi) the Factoring Agreements of such Borrower with the Factor obligated on such Factor Receivable shall be in full force and effect and each party to the Factoring Agreements with the Factor obligated on such Factor Receivable shall be in compliance with the terms and conditions thereof and no breach of such terms or default or event of default thereunder shall exist or have occurred; (vii) the Factor obligated in respect of such Factor Receivable shall not have sent any notice of default or of the failure of such Borrower to comply with any of the terms of the applicable Factoring Agreements or otherwise notified any Borrower or Guarantor of the intention of such Factor to cease or suspend payments to such Borrower in respect of the Factor Receivable; (viii) there are no facts, events or occurrences which would impair the validity, enforceability or collectability of such Factor Receivable or delay payment thereunder; (ix) the Accounts of such Borrower sold giving rise to such Factor Receivable satisfy all of the conditions for the purchase thereof by the Factor obligated thereon and the approval and acceptance of such sale by such Factor in accordance with the terms of the applicable Factoring Agreements and shall not be subject to any chargeback or other right of such Factor to reassign such Account to such Borrower

4

(whether or not such Factor exercises such right) or obligation of such Borrower to pay the amount of such Account to such Factor, whether because the goods sold giving rise to such Account have been rejected or returned by the account debtor owing such Account or otherwise; (x) the Accounts sold giving rise to such Factor Receivable are not unpaid more than the earlier of sixty (60) days after the original due date for them or ninety (90) days after the date of the original invoice for them; (xi) such Accounts sold giving rise to such Factor Receivable comply with the terms and conditions contained in Section 7.2(b) of this Agreement.

The criteria for Eligible Factor Receivables set forth above may only be changed and any new criteria for Eligible Factor Receivables may only be established by Agent in good faith based on either: (1) an event, condition or other circumstance arising after the date hereof, or (2) an event, condition or other circumstance existing on the date hereof to the extent Agent has no written notice thereof from a Borrower prior to the date hereof, in either case under clause (i) or (ii) which adversely affects or could reasonably be expected to adversely affect the Factor Receivables in the good faith determination of Agent. Any Factor Receivables which are not Eligible Factor Receivables shall nevertheless be part of the Collateral.

(d) The definition of “Excess Availability” in Section 1.40 of the Loan Agreement is hereby is hereby deleted in its entirety and the following substituted therefor:

“1.40 “Excess Availability” shall mean, as to each Borrower, the amount, as determined by Agent, calculated at any date, equal to: i) the lesser of: (1) the Borrowing Base of such Borrower and (2) the Loan Limit of such Borrower (in each case under (i) or (ii) after giving effect to any Reserves other than any Reserves in respect of Letter of Credit Accommodations plus the Supplemental Availability Amount), minus ii) the sum of: (1) the amount of all then outstanding and unpaid Obligations of such Borrower (but not including for this purpose Obligations of such Borrower arising pursuant to any guarantees in favor of Agent and Lenders of the Obligations of the other Borrowers or the then outstanding Letter of Credit Accommodations), plus (2) the amount of all Reserves then established in respect of Letter of Credit Accommodations, plus (3) the aggregate amount of all then outstanding and unpaid trade payables and other obligations of such Borrower which are outstanding more than sixty (60) days past due as of such time (other than trade payables or other obligations being contested or disputed by such Borrower in good faith); provided, that, solely for the purposes of determining Quarterly Average Excess Availability in connection with the calculation of the Applicable Margin during any fiscal quarter, Excess Availability shall be calculated without regard to the Loan Limit of any Borrower at Borrower Agent’s option for any monthly period within the fiscal quarter (the “Enhanced Applicable Margin Computation”); provided, further, that; (w) the allowed

5

amount of Quarterly Excess Availability (as used for the purpose of calculation for the Enhanced Applicable Margin) that exceeds the Loan Limit is limited to $75,000,000, (x) Borrower Agent shall have delivered written notice to Agent, not less than five (5) Business Days prior to the end of the fiscal quarter with respect to which Borrower Agent elects the use of the Enhanced Applicable Margin Computation, which notice shall be irrevocable and shall specify that Borrower Agent elects the use of the Enhanced Applicable Margin Computation and (y) Borrowers shall pay to Agent, for the ratable benefit of the Lenders, a fee in respect of the use of the Enhanced Applicable Margin Computation equal to $15,000 for each month that Borrower Agent has elected the use of the Enhanced Applicable Margin Computation, which fee shall be payable on the last day of the effective fiscal quarter and may be charged to any loan account of Borrowers maintained with Agent.”

(e) The definition of “Inventory Loan Limit” in Section 1.65 of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“1.65 “Inventory Loan Limit” shall mean, as to each Borrower, at any time, the amount equal to sixty (60%) percent of the Maximum Credit minus the then outstanding principal amount of Loans to the other Borrowers based on Eligible Inventory (and including Letter of Credit Accommodations to the extent provided in the definition of the term Borrowing Base).”

(f) The definition of “Maximum Credit” in Section 1.80 of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“1.80 “Maximum Credit” shall mean $125,000,000 as such amount may be increased pursuant to and in accordance with the terms of Section 2.4 hereof.”

(g) The definition of “Prime Rate” in Section 1.91 of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“1.91 “Prime Rate” shall mean, on any date, the greater of (a) the rate from time to time publicly announced by Wachovia, or its successors, as its prime rate, whether or not such announced rate is the best rate available at such bank or (b) the Federal Funds Rate in effect on such day plus one-half (  ¹ / 2 %) percent.”

2. Syndication Agent . The Preamble to the Loan Agreement is hereby amended by adding Bank of America, N.A. as Syndication Agent.

3. Loans .

3.1 Inventory Loan Limit . Section 2.1(b)(iii) of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

(iii) the aggregate principal amount of the Loans outstanding at any time to Borrowers based on the Eligible Inventory of Borrowers (and including the then undrawn amounts of Letter of Credit Accommodations used to purchase Inventory to the extent set forth in Section 1.10 hereof) shall not exceed the Inventory Loan Limit.

6

3.2 Letter of Credit Accommodations.

(a) Section 2.2(b) of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“(b) In addition to any charges, fees or expenses charged by any bank or issuer in connection with the Letter of Credit Accommodation, Borrowers shall pay to Agent, for the benefit of Lenders a letter of credit fee at a rate equal to (i) in connection with the daily outstanding balance of the Letter of Credit Accommodations up to $5,000,000, one and one half (1.50%) percent per annum for the immediately preceding month (or part thereof), payable in arrears as of the first day of each succeeding month and (ii) in connection with the outstanding balance of all remaining Letter of Credit Accommodations, the Applicable Eurodollar Rate Margin (on a per annum basis)set forth in Section 1.6 for the immediately preceding month (or part thereof), payable in arrears as of the first day of each succeeding month, except that Agent may, and upon the written direction of Required Lenders shall, require Borrowers to pay to Agent for the benefit of Lenders such letter of credit fee at a rate equal to the then effective applicable letter of credit fee rate plus two (2%) percent per annum in connection with the daily outstanding balance of the Letter of Credit Accommodations for: (A) the period from and after the date of termination hereof until Agent and Lenders have received full and final payment of all Obligations (notwithstanding entry of a judgment against any Borrower) and (B) the period from and after the date of the occurrence of an Event of Default for so long as such Event of Default is continuing as determined by Agent. Such letter of credit fee shall be calculated on the basis of a three hundred sixty (360) day year and actual days elapsed and the obligation of Borrowers to pay such fee shall survive the termination of this Agreement.”

(b) Section 2.2(e) of the Loan Agreement is hereby amended by deleting the reference therein to “$60,000,000” and substituting “$40,000,000” therefor.

3.3 Increases in Maximum Credit . The Loan Agreement is hereby amended by adding the following new Section 2.4:

“2.4 Increases in Maximum Credit .

(a) Borrower (or Borrower Agent on behalf of such Borrower) may, at any time, request that Agent to increase the Maximum Credit. Any such written request shall specify the amount of the increase in the Maximum Credit that

7

Borrowers are requesting; provided, that, (i) the aggregate amount of any such incremental increases in the Maximum Credit (the “Incremental Facility”) does not cause the Maximum Credit to exceed $200,000,000, (ii) such request shall be for an increase shall be in increments of $25,000,000 not to exceed $75,000,000 at any one time or in the aggregate during the term hereof, (iii) any such request shall be irrevocable, except if the amount of increase in Commitments from Lenders is less than the amount requested by Borrowers, then Borrowers shall, in their sole discretion, be entitled to withdraw any such request and no fee related to such request shall be payable hereunder and (iv) in no event shall more than one such written request to increase the Maximum Credit be delivered to Agent in any calendar quarter .

(b) Upon the receipt by Agent of any such written request to increase the Maximum Credit, Agent shall notify each of the Lenders of such request and each Lender shall have the option (but not the obligation) to increase the amount of its Commitment by its Pro Rata Share of the amount of the increase in the Maximum Credit requested by Borrowers as set forth in the notice from Agent to such Lender. Each Lender shall notify Agent within ten (10) days after the receipt of such notice from Agent whether it is willing to so increase its Commitment, and (ii) no Lender shall be obligated to provide such increase in its Commitment and the determination to increase the Commitment of a Lender shall be within the sole and absolute discretion of such Lender. If the aggregate amount of the increases in the Commitments received from the Lenders does not equal or exceed the amount of the increase in the Maximum Credit requested by Borrowers, Agent may seek additional increases from Lenders or Commitments from such Eligible Transferees as it may determine, after consultation with Borrowers. In the event Lenders (or Lenders and any such Eligible Transferees, as the case may be) have committed in writing to provide increases in their Commitments or new Commitments in an aggregate amount in excess of the increase in the Maximum Credit requested by Borrowers or permitted hereunder, Agent shall then have the right to allocate such commitments, first to Lenders and then to Eligible Transferees, in such amounts and manner as Agent may determine, after consultation with Borrowers. If, in connection with the arrangement of additional Commitments for the Incremental Facility the resulting pricing with respect to any Loans or Letters of Credit under the Incremental Facility would be greater than otherwise applicable to Loans or Letters of Credit immediately prior to the Maximum Credit Increase Effective Date, then on and after the Maximum Credit Increase Effective Date, each Interest Rate under this Agreement shall be automatically increased such that in no event shall any Interest Rate applicable to Loans and Letters of Credit in respect of the Incremental Facility exceeds those applicable to other Loans and Letters of Credit hereunder.

(c) The Maximum Credit shall be increased by the amount of the increase in Commitments from Lenders or new Commitments from Eligible Transferees, in each case selected in accordance with this Section 2.4, for which Agent has

8

received Assignment and Acceptances within sixty (60) days after the date of the request by Borrowers for the increase or such earlier date as Agent and Borrowers may agree (but subject to the satisfaction of the conditions set forth below), whether or not the aggregate amount of the increase in Commitments and new Commitments, as the case may be, equal or exceed the amount of the increase in the Maximum Credit requested by Borrowers in accordance with the terms hereo