In this article, I explain and sum up everything one needs to know to apply for a J-1 visa in the US, in order to do an internship. Everything below is based on my experience and on the instructions I received.

Visa

Getting a J-1 visa for the US is a quite long process. First and foremost, a sponsor is required. The most popular sponsor of J-1 visa programs is undoubtedly CIEE, a non-profit organization. But there are others.

CIEE has many partners throughout the world. A good one in France is Parenthèse. It is the one I decided to go with, since my internship company is used to working with them.

Parenthèse's staff is very professional and helpful. They guided me all along the entire process.

However, should you decide to deal with CIEE directly, with no intermediary, you will find everything you need at www.ciee.org/internship-usa.

If my memory serves, the procedure is as follows:

First Parenthèse emailed me a bunch of files to read and fill out (a very long one addressed to CIEE, and a short one for Parenthèse). They also ask for a copy of your passport. Fortunately, you can send them back over email.
Next you pay (roughly 1500€ in total).
By the time they get the money, Parenthèse must have sent your dossier to CIEE. You'll be contacted by them, asking you to provide them with your supervisor's details (on a website). Once done, your supervisor at your company will have to complete a form referred to as DS-7002. Then, you and your supervisor will both have to sign this form electronically, on the same website.
After a few days, Parenthèse will receive a mail with your entire dossier (DS-2019, DS-7002, SEVIS fee payment receipt, proof of insurance, and a letter for the social security). They will forward everything to you.
From now on, it is your responsibility to make an appointment at the US embassy. However you will need to complete an online form (the Online Nonimmigrant Visa Application, aka DS-160) beforehand. Upon completion of this form, you will have to pay a fee (152€ as of December 2016) by making a transfer. Once they receive the money, they will enable you to make an appointment. More information about that on this website. Here is the French website to make an appointment.
Finally, you will get to go to the embassy. Bring your passport, DS-2019, DS-7002, DS-160, SEVIS receipt, a 5"x5" photo of you (although a regular passport photo worked for me), and a proof of ties to your home country. Do not expect anything crazy, you'll get asked a couple of questions (like what the purpose of your trip is), fingerprints scanned, and within minutes you're out! Damn stressing though. Make sure you arrive at least 15 minutes early. Here are two links to articles (written in French), about how to prepare for this interview: Visa J1 – L’ambassade and USA: ENTRETIEN À L'AMBASSADE DES ÉTATS-UNIS D'AMÉRIQUE (the latter is outdated).
Roughly a week later, you will receive your passport with your visa. The last step before leaving is to complete the CIEE online orientation, which is basically an introduction to the US and what it's like to live there. You should normally have received an email with a link to that orientation.

Insurance

Getting a visa with CIEE as a sponsor is quite expensive, but there's a reason for that: they provide you with an insurance plan. Not the best for sure - not every expense is refunded and oftentimes you'll pay copays - but it's sufficient for most people. More information on their website.

First, make sure to carry your insurance card and passport with visa with you at all times.

At the airport, you will be required to show the following documents:

Passport with visa
DS-2019
SEVIS fee payment receipt

Then, there are a few things to do in the next few days. In the correct order, these things include:

Registering for SEVIS by calling CIEE at 1-888-268-6245. Make sure you have your DS2019 form with you when calling since they are going to ask for your number, which begins with "N00". Do not register before the DS-2019 program start date. Provide them with your home address, email address, and American phone number at the same time.
If you don't have a Social Security card, print a copy of your i94 number at www.cbp.gov/i94.
If you don't have a Social Security card, register for a Social Security number at least 2 days after registering for SEVIS and 10 days after entering the US: visit your local social security office (find one on www.ssa.gov/locator). You can visit them without any appointment. There, bring:
- Your DS2019 form
- The letter "Dear social security officer", provided by CIEE
- Your passport with your visa
- Your i94 number (make sure to print the page from the official website, they do not accept any other paper).

After you have applied, the Social Security office will provide you with a "Proof of Application" document. This receipt may enable you to receive paychecks and open a bank account. You should check the status of your application within seven to ten days of submission by visiting a social security office. In case of problem, call CIEE. Then the card will arrive by mail within a few weeks (3 tops, otherwise contact CIEE). This card will be valid for your entire life.

It is also advised to get a state ID card to avoid carrying your passport everywhere.

Likewise, opening a bank account could be useful and sometimes necessary to get paid.

You should notify CIEE of any change in US home address, phone number or email address within 10 days of the change.

During your internship, you'll get two online evaluations to complete, one at mid-point and one at the end of the program. Your supervisor will get online evaluations to complete too. Should there be a problem (you or your supervisor didn't get the emails about the evaluation, for example), email CIEE at evals@ciee.org.

Trips outside of the USA

If you intend to go out of the USA temporarily, call CIEE at least 3 weeks prior to your departure. You will have to do some paperwork and CIEE will need to countersign your DS-2019. Each trip cannot exceed 30 days.

Things to do upon departing

Nothing ("Aliens Not Required To Obtain Sailing or Departure Permits"), as far as I remember. Maybe change your personal details on your bank account, such as your home address.

Taxes

First of all, this article from Parenthèse is quite useful for French expats.

Also, here are some resources as to how to file your tax return (form 1040NR-EZ for J1 visas) if you want to do it by yourself instead of using a third party online service such as TaxBack or TurboTax (TurboTax is not compatible with J1 visa holders):

As to me, well I used TaxBack.com. How did it go? Would I recommend their service? Everything you need to know right below.

TaxBack.com for J1 visa holders

Pros

It's the only online service (that I found) that also handles J1 visas.
They take care of everything, which means you save yourself approximatively a day of hassle.
They customer support is very good. They usually respond to emails within a day.
Their website, although a bit old, is convenient. They offer separate pages for each of your tax refund applications, with a step by step timeline showing you where your file is, in regard to the entire process (from submitting the required files and your W2 form to "you got your money on your bank account"). They handle FICA and US tax applications individually, on distinct pages.
An online chat, very responsive.
They take care of sending all the papers to IRS in the US, in your place.
They can transfer your money back to any bank account in the world, in any currency. I believe the IRS is much more restrictive (only a cheque or transfer in USD on a US bank account?).

Cons

EXPENSIVE: oh boy don't you expect them to be cheap. From the amount I was supposed to get back, they subtracted:

Processing fees
Handling fees
Bank transfer fees ($35 per transfer)

Also they applied a very disadventageous exchange rate (USD to EUR): between May 1 and May 11 (today), the official exchange rate fluctuated between 0.83 and 0.84. They used 0.79052 for my transfer, which occured some day between these two dates. You read it well.

Before using their service, you'd better read this page very carefully. As an example, I was supposed to get $1156.91 back from FICA. The processing fee was $231.38 and the handling fee was $46.28. They kept 24% of my refund. Plus a $35 bank transfer fee. Geez! Not sure my time is worth that much money, after all. Maybe I should have taken the time to fill the documents by myself...

Taxes on bank interest for nonresident aliens

This topic is still quite unclear to me, but apparently it's "nontaxable and nonreportable", as per Federal Income Tax Withholding and Reporting on Other Kinds of U.S. Source Income Paid to Nonresident Aliens. Even when you're no longer living in the US.

This other document says:

"My only income was some bank interest. Do I have to file a tax return?"

"A: If you are a nonresident alien, bank interest is excluded from your U.S. income under a special tax rule intended to encourage foreign investment in the United States. "

Good to know: banks and credit unions send every year in January a document (FORM 1099) that lists earned interest only if above $10.

Also bear in mind that some credit unions refer to interest as dividends:

How to Report Dividends from a Credit Union Account
Difference between dividend and interest
1099-DIV Dividend Income: "Certain distributions commonly referred to as dividends actually should be reported as interest"

Finally, a few additional links related to the topic:

That's it! Hope it was helpful.