ERROR Identifying TIRs - swifter

[diriano]

Hi,
We are running EDTA 2.2.2 on some Citrus genomes, and got the following error during the TIR identification:

Traceback (most recent call last):
  File "/home/user/miniconda3/envs/EDTA2.2/lib/python3.12/site-packages/swifter/swifter.py", line 419, in apply
    tmp_df = func(sample, *args, **kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/EDTA2.2/share/TIR-Learner3/bin/get_fasta_sequence.py", line 17, in <lambda>
    df["end"] = df.swifter.progress_bar(flag_verbose).apply(lambda x: min(x["end"], fasta_len_dict[x["seqid"]]), axis=1)

EDTA continues, but the TIR directory does not have all result files, and the pipeline complains about some missing file sunder the TIR directory:

KeyError: 'CM039161.1_split_1of5'
Can't open ./TIR-Learner-Result/TIR-Learner_FinalAnn.fa: No such file or directory at /home/user/programs/EDTA/bin/rename_tirlearner.pl line 19.
Warning: LOC list GCA_022201045.1_DVS_A1.0_genomic.fna.mod.TIR.ext30.list is empty.

Error: Error while loading sequence
Filter sequence based on TEsorter classifications. Unclassified sequences will also be output to the clean file.
	Usage: perl cleanup_misclas.pl sequence.fa.rexdb.cls.tsv
	Author: Shujun Ou (shujun.ou.1@gmail.com) 10/11/2019
	
mv: cannot stat 'GCA_022201045.1_DVS_A1.0_genomic.fna.mod.TIR.ext30.fa.pass.fa.dusted.cln.cln': No such file or directory
cp: cannot stat 'GCA_022201045.1_DVS_A1.0_genomic.fna.mod.TIR.ext30.fa.pass.fa.dusted.cln.cln.list': No such file or directory
cp: cannot stat 'GCA_022201045.1_DVS_A1.0_genomic.fna.mod.TIR.intact.raw.fa.anno.list': No such file or directory
Can't open ./TIR-Learner-Result/TIR-Learner_FinalAnn.gff3: No such file or directory.
ERROR: No such file or directory at /home/user/programs/EDTA/bin/output_by_list.pl line 39.
Error: TIR results not found!

cat: GCA_022201045.1_DVS_A1.0_genomic.fna.mod.Helitron.intact.raw.bed: No such file or directory

Here are the version of some of the python (3.12.8) modules I think may be relevant:

>>> import swifter
>>> print(swifter.__version__)
1.4.0
>>> import numpy
>>> print(numpy.__version__)
2.2.3
>>> import pandas
>>> print(pandas.__version__)
2.2.2

EDTA was installed using the YML file: https://raw.githubusercontent.com/oushujun/EDTA/refs/heads/master/EDTA_2.2.x.yml and there were no problems during installation.

I replaced line 17 of file share/TIR-Learner3/bin/get_fasta_sequence.py with the following line:

df["end"] = df.swifter.progress_bar(flag_verbose).apply(lambda x: min(x["end"], fasta_len_dict.get(x["seqid"], float('inf'))), axis=1)

And EDTA could complete without problems producing TIR results.
I would appreciate your advice on whether this is a good approach or if you have any alternative suggestions.
Thanks in advance, and I truly appreciate the great software!

Best,
Diego

[chloee-mclaughlin]

Following this thread as I am running into the same error message. We are using EDTA2.2.2 downloaded from the most recent GitHub update, created with conda. Thank you in advance!

Best,
Chloee

Species: others
Traceback (most recent call last):
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/lib/python3.12/site-packages/swifter/swifter.py", line 419, in apply
    tmp_df = func(sample, *args, **kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/share/TIR-Learner3/bin/get_fasta_sequence.py", line 23, in <lambda>
    df["end"] = df.swifter.progress_bar(flag_verbose).apply(lambda x: min(x["end"], fasta_len_dict[x["seqid"]]), axis=1)
                                                                                    ~~~~~~~~~~~~~~^^^^^^^^^^^^
TypeError: unhashable type: 'Series'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/share/TIR-Learner3/TIR-Learner.py", line 95, in <module>
    TIRLearner_instance = TIRLearner(genome_file, genome_name, species, TIR_length,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/share/TIR-Learner3/bin/main.py", line 91, in __init__
    self.execute()
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/share/TIR-Learner3/bin/main.py", line 131, in execute
    self.execute_M4()
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/share/TIR-Learner3/bin/main.py", line 632, in execute_M4
    self["base"] = get_fasta_sequence.execute(self)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/share/TIR-Learner3/bin/get_fasta_sequence.py", line 67, in execute
    df = get_start_end(TIRLearner_instance.genome_file_path, TIRLearner_instance["base"],
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/share/TIR-Learner3/bin/get_fasta_sequence.py", line 23, in get_start_end
    df["end"] = df.swifter.progress_bar(flag_verbose).apply(lambda x: min(x["end"], fasta_len_dict[x["seqid"]]), axis=1)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/lib/python3.12/site-packages/swifter/swifter.py", line 428, in apply
    timed = timeit.timeit(wrapped, number=N_REPEATS)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/lib/python3.12/timeit.py", line 237, in timeit
    return Timer(stmt, setup, timer, globals).timeit(number)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/lib/python3.12/timeit.py", line 180, in timeit
    timing = self.inner(it, self.timer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<timeit-src>", line 6, in inner
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/lib/python3.12/site-packages/swifter/swifter.py", line 337, in wrapped
    self._obj.iloc[self._SAMPLE_INDEX].apply(
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/lib/python3.12/site-packages/pandas/core/frame.py", line 10374, in apply
    return op.apply().__finalize__(self, method="apply")
           ^^^^^^^^^^
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/lib/python3.12/site-packages/pandas/core/apply.py", line 916, in apply
    return self.apply_standard()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/lib/python3.12/site-packages/pandas/core/apply.py", line 1063, in apply_standard
    results, res_index = self.apply_series_generator()
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/lib/python3.12/site-packages/pandas/core/apply.py", line 1081, in apply_series_generator
    results[i] = self.func(v, *self.args, **self.kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cmclaughlin/.conda/envs/EDTA2.2/share/TIR-Learner3/bin/get_fasta_sequence.py", line 23, in <lambda>
    df["end"] = df.swifter.progress_bar(flag_verbose).apply(lambda x: min(x["end"], fasta_len_dict[x["seqid"]]), axis=1)
                                                                                    ~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'NC_081805.1_split_1of6'
Can't open ./TIR-Learner-Result/TIR-Learner_FinalAnn.fa: No such file or directory at /home/cmclaughlin/.conda/envs/EDTA/share/EDTA/bin/rename_tirlearner.pl line 19.
Warning: LOC list GCF_030704535.1_ASM3070453v1_genomic_reheader.fna.mod.TIR.ext30.list is empty.

Error: Error while loading sequence
Filter sequence based on TEsorter classifications. Unclassified sequences will also be output to the clean file.
	Usage: perl cleanup_misclas.pl sequence.fa.rexdb.cls.tsv
	Author: Shujun Ou (shujun.ou.1@gmail.com) 10/11/2019
	
mv: cannot stat 'GCF_030704535.1_ASM3070453v1_genomic_reheader.fna.mod.TIR.ext30.fa.pass.fa.dusted.cln.cln': No such file or directory
cp: cannot stat 'GCF_030704535.1_ASM3070453v1_genomic_reheader.fna.mod.TIR.ext30.fa.pass.fa.dusted.cln.cln.list': No such file or directory
cp: cannot stat 'GCF_030704535.1_ASM3070453v1_genomic_reheader.fna.mod.TIR.intact.raw.fa.anno.list': No such file or directory
Can't open ./TIR-Learner-Result/TIR-Learner_FinalAnn.gff3: No such file or directory.
ERROR: No such file or directory at /home/cmclaughlin/.conda/envs/EDTA/share/EDTA/bin/output_by_list.pl line 39.
Error: TIR results not found!

ERROR: Raw TIR results not found in GCF_030704535.1_ASM3070453v1_genomic_reheader.fna.mod.EDTA.raw/GCF_030704535.1_ASM3070453v1_genomic_reheader.fna.mod.TIR.intact.raw.fa
	If you believe the program is working properly, this may be caused by the lack of intact TIRs in your genome. Consider to use the --force 1 parameter to overwrite this check