.gain reference file not usable

Sebastien Moniot

unread,

Feb 26, 2021, 5:10:10 AM2/26/21

to Warp

Hi all,

I recently obtained a cryo-EM dataset collected on a Falcon 4 detector and thus with .eer files.

Warp handles them fine but I cannot get to use the reference.gain file for preprocessing. Warp only accepts .dm4, .mrc, or .em extensions.

I tried to replace the extension of my .gain file to the others without success, Warp then complaining that the file cannot be read.

Any Idea how to solve my problem?

Thanks.

Sebastien

Takanori Nakane

unread,

Feb 26, 2021, 5:21:32 AM2/26/21

to Warp

Hi,

An EER gain reference with an .gain extension is internally a TIFF file.

An important point is that Warp (and RELION) divide raw detector values by EER's gain reference in the MRC format.

This is the convention used by earlier versions of the Falcon4 controller. In contrast to this, a newer gain reference in the TIFF container

is a multiplicative gain, as in K2 and K3. That is, the raw values must be multiplied by the gain reference.

So, what you should do is:

1. convert .gain into .mrc

2. take the reciprocal of the file

Both can be done with e2proc2d.py in EMAN2.

Best regards,

Takanori Nakane

Florian Roßmann

unread,

Oct 20, 2021, 4:35:56 PM10/20/21

to Warp

Hi,

Sorry to open this up again. We followed these steps from Takanori, but we still see vertical lines on my averaged images. Preprocessing of the same .eer frames works fine in cryoSPARC(Live) and RELION.

Any ideas what else we could try?

Thanks,

Flo

Takanori Nakane

unread,

Oct 20, 2021, 6:51:01 PM10/20/21

to Warp

Hi,

Are your vertical lines very sharp? If so, they came from dead lines on your detector.

These pixels are marked by 0 in the gain reference and RELION tries to conceal them

by replacing them with random values taken from neighboring good pixels.

I don't know if Warp has such a defect correction algorithm.

Best regards,

Takanori Nakane

Florian Roßmann

unread,

Oct 20, 2021, 7:26:48 PM10/20/21

to Warp

Hi Takanori,

Thanks for the quick reply. Yes, it is a pixel defect (3 pixels in a row) and it was also concealed in the camera software, which is why the line does not appear in mrc files. I thought the gain ref would take care of it...

Flo

Takanori Nakane

unread,

Oct 20, 2021, 7:50:53 PM10/20/21

to Warp

Hi,

Dead pixels have the value of zero, so you cannot conceal them by multiplying them with anything...

I keep saying TFS that the defects should be dealt with by the camera software before writing a file,

because the best correction algorithm depends on the details of the camera hardware and can

vary from model to model. It is not practical for RELION, Warp etc to implement different algorithms for

every new camera.

Best regards,

Takanori Nakane

Max Renner

unread,

Feb 23, 2022, 9:17:29 AM2/23/22

to Warp

Hi,

>So, what you should do is:

>1. convert .gain into .mrc

>2. take the reciprocal of the file

>Both can be done with e2proc2d.py in EMAN2.

Sorry if this is obvious, but what is the command to take the reciprocal in e2proc2d.py?

Thanks! :)

3P1L

unread,

Mar 30, 2022, 3:32:55 AM3/30/22

to Warp

Dear All,

is there any workable solution? In fact, I tried all of the above, but the vertical lines remain.

I've tried making a separate defects file from the initial .gain, too. I guess 0s are dead pixels?

(My current workaround is using Relion's implementation)

Maybe we can find a solution and archive it here for posterity! ;-)

Best,

Philipp.

Zhengyi Yang

unread,

Apr 27, 2022, 5:37:32 AM4/27/22

to Warp

I'm still looking for the best way to process the .eer and .gain file.

There are two ways that work for me, but I'd like some people to confirm as well.

1. convert the .gain file to .mrc, either use newstack -in *gain -ou gain.mrc, or simply rename the suffix should do the job.

2. find the reciprocal. one way is the relion_estimate_gain, since it only takes star file as input, you will need to create a relion project and import the above gain.mrc.

alternatively, you can open the gain.mrc in FIJI, Process -> Math -> reciprocal, then save it.

I found neither way elegant. so any other suggestion would be very welcome.

Max Renner

unread,

May 4, 2022, 10:08:48 AM5/4/22

to Warp

Hi Zhengy and Philipp,

This is what I have been doing to prepare the gain ref for Warp, starting from a .gain file:

tif2mrc GainReference.gain gain.mrc

e2proc2d.py --process math.reciprocal gain.mrc gain_reciprocal.mrc

I got ~4.7A for STA from a small set of ribosome tilt series (using WARP-RELION-M) using this gain ref, so I think it should be ok...

Hope that helps :)

Max

Zhengyi Yang

unread,

May 5, 2022, 8:17:09 AM5/5/22

to Warp

Dear Max, Thanks for the reply. For anyone's information, now there is a very good explanation on relion documentation.

The confusion here is because historically Falcon is producing .eer files with .mrc gain reference. To apply gain reference, the .eer needs to be divided by the .mrc, which is the opposite of what you need to do with K2/K3 raw tiffs (multiplying).

The .gain reference is a recent development, which you only need to multiply to the raw .eer, so similar to K2/K3, which should make our life easier, except Warp doesn't accept .gain format yet. If you convert .gain to .mrc, Warp thinks this is the old version that it needs dividing, hence giving you the wrong result.

Best,

Zhengyi

3P1L

unread,

Jun 16, 2023, 1:54:28 PM6/16/23

to Warp

Hi @All!

Just to come back to this ... I am not sure if the above is still true ...

At least not for eer data with a falcon4i (recorded with the newest SerialEM) and Warp 1.1 beta.

tif2mrc + 1/gain does not look OK -> see attached image

tif2mrc looks good

Does anybody know if the .gain was changed (to multiplicative), or does Warp handle it differently now?

🤷‍♂️

Philipp.

4warp.jpg

Jinyi Zhu

unread,

Jun 17, 2023, 2:26:06 PM6/17/23

to Warp

Hi Philipp,

If you use the commands from Max Renner to generate gain.mrc, I would suggest you try to:

1) flip y axis for the gain in warp 1.1 beta,

2) turn on "use movie sum" in CTF section.

My data were captured by EPU directly, so I am not sure if those apply to your SerialEM collected data

It is odd for flip Y axis as the same .gain file was used in Relion 4 and CryoSparc without any flipping and never caused any trouble. However, without flipping, I always see the bright vertical lines.

Jinyi

3P1L

unread,

Jun 17, 2023, 3:36:55 PM6/17/23

to Warp

Hey Jinyi,

thanks. The flipping does remove the bright line.

Philipp.

Tom

unread,

Sep 22, 2023, 5:41:06 AM9/22/23

to Warp

Hi all,

this is a bit of a different topic, but to all I see for now, it relates to the gain reference inversion. I am seeing small white rectangular patches in my gain corrected .eer files (I used the eman2 commands to convert/invert the gain file). They translate to white stripes orthogonal to the image plane, after alignment. Did anyone see this in their data (and could find a reason/fix for it)?

Best,

Tom

Reply all

Reply to author

Forward