NetCDF: Access failure (NetCDF error code: -77) on OPeNDAP

[natgeo-wong]

Hi!

I'm having some trouble with using GitHub Actions to create documentation when access Earthdata OPeNDAP servers (see the code blocks (https://georegionsecosystem.github.io/NASAPrecipitation.jl/dev/using/download)[here]).

In the runner log, it gives me the following error:

exception = 
    NetCDF error: NetCDF: Access failure (NetCDF error code: -77)
    Stacktrace:
      [1] check
        @ ~/.julia/packages/NCDatasets/xJGTj/src/errorhandling.jl:25 [inlined]
      [2] nc_get_vars!(ncid::Int32, varid::Int32, startp::Vector{Int64}, countp::Vector{Int64}, stridep::Vector{Int64}, ip::Matrix{Float32})
        @ NCDatasets ~/.julia/packages/NCDatasets/xJGTj/src/netcdf_c.jl:1048

and so on. I've tried googling what exactly is wrong, but I'm not too sure. I can open the dataset to see the metadata information, but I can't access the data itself.

So basically, I can do

hroot = "https://gpm1.gesdisc.eosdis.nasa.gov/opendap/AUXILIARY"
npdnc = "GPM_IMERG_LandSeaMask.2/GPM_IMERG_LandSeaMask.2.nc4"
npdds = NCDataset(joinpath(hroot,npdnc),"r")

But if I try to access the data via npdds["landseamask"][1], the data cannot be access.

Is this a GitHub action error? Or is there something else I'm not seeing?

This is not a critical failure, since I can get the code to work on multiple platforms, macOS and Linux alike. I just can't get it to work on GitHub actions, but I would like to eventually because this error means I cannot showcase examples in the documentation.

[Alexander-Barth]

Did you set up the .netrc on GitHub ?

I do have one with an entry urs.earthdata.nasa.gov (with a password still accepted at earth data login page) in .netrc and I also added a .dodsrc as you explain.

But NCDatasets also fails for me:

julia> npdds["landseamask"][1]
ERROR: NetCDF error: NetCDF: Malformed or inaccessible DAP2 DATADDS or DAP4 DAP response (NetCDF error code: -73)
Stacktrace:
 [1] check
   @ ~/.julia/dev/NCDatasets/src/errorhandling.jl:25 [inlined]
 [2] nc_get_vars!(ncid::Int32, varid::Int32, startp::Vector{Int64}, countp::Vector{Int64}, stridep::Vector{Int64}, ip::Matrix{Float32})
   @ NCDatasets ~/.julia/dev/NCDatasets/src/netcdf_c.jl:1048
 [3] _read_data_from_nc!(::NCDatasets.Variable{Float32, 2, NCDataset{Nothing, Missing}}, ::Matrix{Float32}, ::UnitRange{Int64}, ::Vararg{UnitRange{Int64}})
   @ NCDatasets ~/.julia/dev/NCDatasets/src/variable.jl:407
 [4] readblock!
   @ ~/.julia/dev/NCDatasets/src/variable.jl:395 [inlined]
 [5] getindex_disk(a::NCDatasets.Variable{Float32, 2, NCDataset{Nothing, Missing}}, i::Int64)
   @ DiskArrays ~/.julia/packages/DiskArrays/WgDSb/src/indexing.jl:54
 [6] getindex
   @ ~/.julia/packages/DiskArrays/WgDSb/src/indexing.jl:312 [inlined]
 [7] getindex(v::CommonDataModel.CFVariable{Union{…}, 2, NCDatasets.Variable{…}, CommonDataModel.Attributes{…}, @NamedTuple{…}}, indexes::Int64)
   @ CommonDataModel ~/.julia/dev/CommonDataModel/src/cfvariable.jl:452
 [8] top-level scope
   @ REPL[12]:1
Some type information was truncated. Use `show(err)` to see complete types.

(@v1.11) pkg> st NetCDF_jll
Status `/mnt/data1/abarth/.julia/environments/v1.11/Project.toml`
  [7243133f] NetCDF_jll v401.900.300+0

Surprisingly, also ncdump (v4.9.2) fails:

ncdump -v lon https://gpm1.gesdisc.eosdis.nasa.gov/opendap/AUXILIARY/GPM_IMERG_LandSeaMask.2/GPM_IMERG_LandSeaMask.2.nc4

		:Title = "Land/Sea static mask relevant to IMERG precipitation 0.1x0.1 degree" ;
		:Version = "2" ;
		:History = "This land sea mask originated from the NOAA group at SSEC in the 1980s. It was originally produced at 1/6 deg resolution, and then regridded for the purposes of GPCP, TMPA, and IMERG precipitation products. NASA code 610.2, Global Change Data Center, restructured this IMERG land sea mask to match the IMERG grid, and converted the file to CF-compliant netCDF4. Version 2 was created in May, 2019 to resolve detected inaccuracies in coastal regions." ;
		:DOI = "10.5067/6P5EM1HPR3VD" ;
data:

syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: HTTP^ Basic: Access denied.
NetCDF: Access failure
Location: file ?; fcn ? line 478

It seems that I am not allowed to access this data.

[natgeo-wong]

That's ... very interesting. I haven't had this particular error in a very long while and often it depends on connection (I do remember this syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR, from when I was in the Azores, but it went back to normal in the US).

[natgeo-wong]

Could you try using an .ncrc file as well, with the same inputs as .dodsrc since from what I understand .ncrc is outdated?

[Alexander-Barth]

Actually, I have the same error (tested with ncdump)

[natgeo-wong]

When you say using ncdump, you mean you can't access the data even when you're not using Julia?

[Alexander-Barth]

yes, exactly. The error is raised in the C code also when not using julia.

[natgeo-wong]

Oh interesting, so you're basically unable to access the data at all even if you follow the instructions here:

https://docs.opendap.org/index.php/DAP_Clients_-_Authentication#Earth_Data_Login_(URS)

[Alexander-Barth]

yes, this is true (I get the error syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR context: HTTP^ Basic: Access denied. with these settings.

$ cat .dodsrc 
HTTP.COOKIEJAR=/home/abarth/.urs_cookies
HTTP.NETRC=/home/abarth/.netrc

with credentials in .netrc.