Invalid literal for int: error while solving MPS files with longer column names

[siddharth-krishna]

I'm facing an error when running linopy on MPS files where column names have more than one letter in them (e.g. col01 as opposed to x01). Here is an example MPS file:

NAME        sample_mip
ROWS
 N  obj     
 G  c1      
 L  c2      
 E  c3      
COLUMNS
    col1        obj       5
    col1        c1        2
    col1        c2        4
    col1        c3        1
    MARK0000  'MARKER'                 'INTORG'
    col2        obj       3
    col2        c1        3
    col2        c2        2
    col2        c3        1
    col3        obj       7
    col3        c1        4
    col3        c2        3
    col3        c3        1
    MARK0001  'MARKER'                 'INTEND'
RHS
    RHS_V     c1        12
    RHS_V     c2        15
    RHS_V     c3        6
BOUNDS
 UP BOUND     col1        4
 UI BOUND     col2        3
 UI BOUND     col3        5
ENDATA

I run this using the python script

import sys
from pathlib import Path
from linopy import solvers

s = solvers.Highs()
result = s.solve_problem(problem_fn=Path(sys.argv[1]), solution_fn=Path("/tmp/sid-linopy-out.sol"))
print(result)
print(result.solution.objective)

Which results in the following error (a similar error appears when using GLPK/Gurobi solvers):

Traceback (most recent call last):
  File ".../linopy-test.py", line 6, in <module>                                           
    result = s.solve_problem(problem_fn=Path(sys.argv[1]), solution_fn=Path("/tmp/sid-linopy-out.sol"))                       
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                       
  File ".../lib/python3.12/site-packages/linopy/solvers.py", line 299, in solve_problem
    return self.solve_problem_from_file(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                      
  File ".../lib/python3.12/site-packages/linopy/solvers.py", line 775, in solve_problem_from_file
    return self._solve(                                                                                                       
           ^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/linopy/solvers.py", line 867, in _solve    
    solution = self.safe_get_solution(status=status, func=get_solver_solution)                                                
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                
  File ".../lib/python3.12/site-packages/linopy/solvers.py", line 224, in safe_get_solution
    return func()                                                                                                             
           ^^^^^^
  File ".../lib/python3.12/site-packages/linopy/solvers.py", line 860, in get_solver_solution
    ).pipe(set_int_index)
      ^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/pandas/core/generic.py", line 6231, in pipe
    return common.pipe(self, func, *args, **kwargs)                                                                           
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/pandas/core/common.py", line 502, in pipe  
    return func(obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/linopy/solvers.py", line 138, in set_int_index
    series.index = series.index.str[1:].astype(int)                                                                           
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                           
  File ".../lib/python3.12/site-packages/pandas/core/indexes/base.py", line 1097, in astype
    new_values = astype_array(values, dtype=dtype, copy=copy)                                                                 
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/pandas/core/dtypes/astype.py", line 182, in astype_array
    values = _astype_nansafe(values, dtype, copy=copy)                                                                        
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                        
  File ".../lib/python3.12/site-packages/pandas/core/dtypes/astype.py", line 133, in _astype_nansafe
    return arr.astype(dtype, copy=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                       
ValueError: invalid literal for int() with base 10: 'ol1'  

I first noticed this error when running on the following JuMP-HiGHS benchmark: Sienna_modified_RTS_GMLC_DA_sys_NetDC_Horizon12_Day332

Could it be that linopy is expecting all column names to look like \w\d+? @FabianHofmann 🙏

[FabianHofmann]

hey sid, you are exactly right! perhaps we need a pre-check on long the prefix is? we could use something like:

def count_initial_letters(word):
    count = 0
    for char in word:
        if char.isalpha():
            count += 1
        else:
            break
    return count

Run that once, just before

linopy/linopy/solvers.py

Line 141 in e5cc814

series.index = series.index.str[1:].astype(int)

I am proposing a change right now

[siddharth-krishna]

Thanks, Fabian, for the super quick response and fix! Btw, does the MPS format guarantee that all columns are named with the same prefix followed by sequential integers?

[FabianHofmann]

@siddharth-krishna I cannot say for sure, I would just say that another format would not make sense. But if we should extend the pre-processing, let me know. If the fix works and you are fine with it, we can merge it it.

[odow]

Btw, does the MPS format guarantee that all columns are named with the same prefix followed by sequential integers?

It most surely does not!!!

See

• https://lpsolve.sourceforge.net/5.5/mps-format.htm
• https://www.ibm.com/docs/en/cofz/12.9.0?topic=formats-working-mps-files

Pretty much any string is allowed. There are two MPS formats, free and fixed. In the free format, anything except ' ' is allowed. (n the fixed format, names are based on the column of each row, and names are theoretically white-space sensitive, except that most optimizers will probably complain.) When writing MPS files, JuMP replaces all spaces with _.

You cannot infer the integer index of a column or row based on the name.

Rows are provided in the order of the ROWS section, and columns are ordered (most commonly) based on their order in the COLUMS section, with the complication that some variables may appear in other sections like BOUNDS without appearing in COLUMNS.

[siddharth-krishna]

Thanks, Oscar. And indeed, I have now seen multiple MPS files from our benchmarks where row and column names are not in such a standardized format.

@FabianHofmann I took a look into the code, and if I'm not mistaken, we use set_int_index to convert the pd.Series index from type string (containing variable/constraint names) to type int. If the input to the solver comes from a linopy.Model, this code works well, because we ensure that vars & constraints look like x1, x2, .... It's only in the case that the solver input comes from an external file that this code breaks.

In this case, I'm wondering if it's okay to reset the index? Essentially throw away the var/constraint names and keep the same order of solution values as returned by the solver. Since the user didn't create the input model in linopy, this order should not conflict with anything. And if the user called the solver on an input file, they will most likely use the output file to obtain the solution, not the linopy.Solution object (which only has an integer ordered solution Series, not a mapping from names to values).

Is my understanding correct? If so, I'm happy to open a PR with the changes above.

(Alternatively, I'm wondering why we want linopy.Solution.primal and dual to have integer indices? Maybe we can generalize them to having string indices too?)

[odow]

I don't know the details of linopy. For your use-case, you don't care about the primal solution so it's okay to loose a mapping between the input file names and the solution in linopy. More generally though, I would expect that linopy could read an MPS file and keep the names.

[FabianHofmann]

I am getting more and more the feeling that the `set_int_index' is actually in the wrong place in the linopy process. If we moved it out of the solver functions to where we actually map the solution back to the linopy model, then the solver functions themselves would be more consistent in the sense that the returned solution would have the same names as the input files (MPS/LP). Let me try that out

[FabianHofmann]

@siddharth-krishna could you try #406 out?