Error: ENOENT: no such file or directory, copyfile 'node_modules/pbf/dist/pbf.js'

[nakkore]

Hi,

I'm using the docker image.

When running docker run gtfs-to-html

it outputs something like:

GTFS-Realtime import complete: 220 alerts, 24882 trip updates, 1254 vehicle positions
Creating DB indexes
Completed GTFS import in 106.2 seconds

Error: ENOENT: no such file or directory, copyfile 'node_modules/pbf/dist/pbf.js' -> '/html/marintransit/js/pbf.js'
  Error: ENOENT: no such file or directory, copyfile 'node_modules/pbf/dist/pbf.  js' -> '/html/marintransit/js/pbf.js'
  
  - promises:623 async copyFile
    node:internal/fs/promises:623:10
  
  - gtfs-to-html.js:2342 async copyStaticAssets
    [lib]/[gtfs-to-html]/dist/bin/gtfs-to-html.js:2342:5
  
  - gtfs-to-html.js:2485 async gtfsToHtml
    [lib]/[gtfs-to-html]/dist/bin/gtfs-to-html.js:2485:5
  
  - gtfs-to-html.js:2607 async setupImport
    [lib]/[gtfs-to-html]/dist/bin/gtfs-to-html.js:2607:3

Is there something wrong with the docker build ?
Tried 2.11.0, 2.10.9, 2.9.10 same issue.

Thanks

[brendannee]

Thanks for reporting this issue and testing various versions.

I just deployed a new version which does a better job of resolving the path to node_modules to copy these files to the output directory:

https://github.com/BlinkTagInc/gtfs-to-html/releases/tag/2.11.1

Let me know if this solves the issue.

[nakkore]

I'm afraid nothing changed, same error.

[brendannee]

Thanks for checking. Can you ensure that the version you are using within docker is updated to 2.11.1 - you should at least get a slightly different error now because the copyFile function has changed?

Can you also update the node.js version in Dockerfile to node:22 - I think this is now the minimum version.

[nakkore]

after changing the node js version it worked but it stopped 20 mins later with the following error:

marintransit: Generating HTML timetables ==============-------------------------- 148/430
marintransit: Generating HTML timetables ==============-------------------------- 149/430
marintransit: Generating HTML timetables ==============-------------------------- 150/430
marintransit: Generating HTML timetables ==============-------------------------- 151/430
marintransit: Generating HTML timetables ==============-------------------------- 152/430
marintransit: Generating HTML timetables ==============-------------------------- 153/430
<--- Last few GCs --->

[1:0x408e0000] 1039027 ms: Mark-Compact 3812.1 (4128.2) -> 3798.6 (4129.7) MB, pooled: 12 MB, 962.97 / 0.81 ms (average mu = 0.167, current mu = 0.071) allocation failure; scavenge might not succeed
[1:0x408e0000] 1040056 ms: Mark-Compact 3814.3 (4129.7) -> 3798.6 (4130.0) MB, pooled: 12 MB, 997.44 / 0.00 ms (average mu = 0.103, current mu = 0.031) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

1: 0xe1803a node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
2: 0x11ec9c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
3: 0x11ecc97 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
4: 0x141a575 [node]
5: 0x141a5a3 [node]
6: 0x143367a [node]
7: 0x1436848 [node]
8: 0x1c9c351 [node]

Also,
it would it be nice to understand what it's actually doing and where it's writing these "HTML timetables". I couldn't find any html files in the html folder.

Thanks

[brendannee]

This is good news as it means the first issue has been resolved and we're on to the next issue.

Docker is running out of memory - can you try updating the docker-compose.yml file as follows to set 8 GB of memory and see if that solves the issue?

version: '3'

services:
  gtfs-to-html:
    image: gtfs-to-html
    environment:
      - NODE_OPTIONS=--max-old-space-size=8192
    build:
      context: .
    volumes:
      - ./html:/html
      - ./config.json:/config.json

[nakkore]

Hi,
with - NODE_OPTIONS=--max-old-space-size=16384

I get:

.......................................................
.........................................................
marintransit: Generating HTML timetables ==============-------------------------- 154/431
marintransit: Generating HTML timetables ==============-------------------------- 155/431
<--- Last few GCs --->

[1:0x44e8b000] 1167802 ms: Mark-Compact 3802.7 (4131.0) -> 3778.4 (4129.0) MB, pooled: 6 MB, 923.53 / 1.55 ms (average mu = 0.382, current mu = 0.423) allocation failure; scavenge might not succeed
[1:0x44e8b000] 1168852 ms: Mark-Compact 3794.4 (4129.2) -> 3776.9 (4125.7) MB, pooled: 9 MB, 984.75 / 1.58 ms (average mu = 0.251, current mu = 0.062) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

1: 0xe36068 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
2: 0x1202550 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
3: 0x1202827 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
4: 0x1430105 [node]
5: 0x1430133 [node]
6: 0x144920a [node]
7: 0x144c3d8 [node]
8: 0x1cb2241 [node]

[brendannee]

Thanks for trying that.

It looks like that didn't allocate any additional memory - can you try updating the Dockerfile with the line ENV NODE_OPTIONS=--max-old-space-size=16384 so that the file looks like:

# syntax=docker/dockerfile:1
FROM node:22

RUN apt update
RUN apt install -y chromium

RUN cd ~/
COPY config.json ./

ENV NPM_CONFIG_PREFIX=/home/node/.npm-global
ENV NODE_OPTIONS=--max-old-space-size=16384
ENV PATH=$PATH:/home/node/.npm-global/bin
RUN npm install -g gtfs-to-html

CMD [ "gtfs-to-html" ]

[nakkore]

after updating the Dockerfile, it seems that it worked:

marintransit: Generating HTML timetables ======================================== 431/431
Warning: Unable to simplify geojson

marintransit: HTML timetables created at /html/marintransit
┌────────────────────────────────────────┬────────────────────┐
│ Item │ Count │
├────────────────────────────────────────┼────────────────────┤
│ 📄 Timetable Pages │ 358 │
├────────────────────────────────────────┼────────────────────┤
│ 🕑 Timetables │ 5937 │
├────────────────────────────────────────┼────────────────────┤
│ 📅 Calendar Service IDs │ 3922 │
├────────────────────────────────────────┼────────────────────┤
│ 🔄 Routes │ 358 │
├────────────────────────────────────────┼────────────────────┤
│ 🚍 Trips │ 1000690 │
├────────────────────────────────────────┼────────────────────┤
│ 🛑 Stops │ 21313 │
├────────────────────────────────────────┼────────────────────┤
│ ⛔ Warnings │ 451 │
└────────────────────────────────────────┴────────────────────┘
marintransit: HTML timetable generation required 2006.6 seconds

However, there's nothing in html/marintransit folder except an empty js folder.

[brendannee]

Glad that we solved the issue of running out of memory.

I think with docker, you need to do the following:

# Get container ID
docker container ls -a

# Copy files (replace with your container ID)
docker cp <container-id>:/html .

What is in the html folder?

Can you share with me your config.json file?

[nakkore]

Hi, you’re right — I missed that in the instructions, sorry about that.

Since the docker-compose.yml includes

volumes:
  - ./html:/html

I’d expect the processing output to land directly in the mapped html folder, rather than needing to copy it manually. Having to move it by hand feels a bit counterintuitive — hope that makes sense.

In the generated log.txt I can see a number of entries similar to:
Timetable 036|undefinedundefinedundefinedundefinedundefinedundefinedundefined|0's trips have stoptimes with timepoints but showOnlyTimepoint is true. Try setting showOnlyTimepoint to false.

But my config does have showOnlyTimepoint=false

Anyway you can check yourself. I'm attaching my config.

config.json

[brendannee]

Thanks for all of your feedback and sharing your config.json

• For the docker output, do you have a suggestion for a better config or way to make the output go directly into the mapped html folder? I'm not a Docker export and don't usually use GTFS-to-HTML from within docker so I'm open to suggestions.

• I checked out the GTFS that you are using in config.json and noticed that it does not use calendar.txt and instead defines each day as an exception in calendar_dates.txt. This was causing the timetable filename to have the "undefined" in it. I pushed an update to GTFS-to-HTML version 2.11.2 https://github.com/BlinkTagInc/gtfs-to-html/releases/tag/2.11.2 which fixes this. Try updating the GTFS-to-HTML version you are using.

[nakkore]

I'm not an expert too, I just know the basics.
However, I suggest you to read https://docs.docker.com/reference/compose-file/volume.
Quickly, considering this

volumes:
      - ./html:/html

your app running in the container should simply write to /html.

I tried 2.11.2 but I get:

Completed GTFS import in 96.0 seconds

Error: ENOENT: no such file or directory, copyfile '/home/node/.npm-global/lib/node_modules/gtfs-to-html/node_modules/pbf/dist/pbf.js' -> '/html/marintransit/js/pbf.js'
  Error: ENOENT: no such file or directory, copyfile '/home/node/.npm-global/lib  /node_modules/gtfs-to-html/node_modules/pbf/dist/pbf.js' -> '/html/marintransi  t/js/pbf.js'
  
  - promises:623 async copyFile
    node:internal/fs/promises:623:10
  
  - gtfs-to-html.js:2364 async copyStaticAssets
    [lib]/[gtfs-to-html]/dist/bin/gtfs-to-html.js:2364:5
  
  - gtfs-to-html.js:2526 async gtfsToHtml
    [lib]/[gtfs-to-html]/dist/bin/gtfs-to-html.js:2526:5
  
  - gtfs-to-html.js:2648 async setupImport
    [lib]/[gtfs-to-html]/dist/bin/gtfs-to-html.js:2648:3

Thanks

[brendannee]

This is a similar issue to the initial one - inside of docker the script is unable to find the node_modules folder.

Can you tell me if the /home/node/.npm-global/lib/node_modules/gtfs-to-html/node_modules/pbf/ folder exists? Or let me know if you can tell where the node modules folder is?

[brendannee]

One workaround for this issue: set noHead to true in your config.json: https://gtfstohtml.com/docs/configuration#nohead

This will skip trying to copy frontend js libraries to the output folder which will let you generate timetables with no header.

[nakkore]

tried with noHead, but I have again the JavaScript heap out of memory.

On other hand, the "generating timetables" process is quite slow (25 mins or even more to barely reach 50%) so I fear that the memory consumption and slowness might be symptoms of something which is not right in this process.