cellxgene_VIP deploying with Heroku

[skim245]

Hi,
I would like to deploy the cellxgene VIP result with Heroku

and would like to know how to modify the "Dockerfile" provided by cellxgene

Here is my modified version and I'm getting this error

---

FROM ubuntu:bionic

ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

RUN apt-get update &&
apt-get install -y build-essential libxml2-dev python3-dev python3-pip zlib1g-dev python3-requests &&
apt-get install git -y
git clone https://github.com/interactivereport/cellxgene_VIP.git

COPY . .

RUN pip install -r requirements.txt

RUN conda env create -n VIP -f VIP.macOS.yml

RUN conda init bash

RUN conda activate VIP

RUN echo "Make sure flask is installed:"
RUN python -c "import flask"

COPY run.py .
RUN ./config.macOS.sh

ENTRYPOINT ["cellxgene"]

---

and getting below errors

E: Unable to locate package clone
E: Unable to locate package https://github.com/interactivereport
E: Couldn't find any package by glob 'https://github.com/interactivereport'
E: Couldn't find any package by regex 'https://github.com/interactivereport'
The command '/bin/sh -c apt-get update && apt-get install -y build-essential libxml2-dev python3-dev python3-pip zlib1g-dev python3-requests && apt-get install git -y git clone https://github.com/interactivereport/cellxgene_VIP.git' returned a non-zero code: 100
▸ Error: docker build exited with Error: 100

Thanks,

[FADHLyemen]

I would love to see how this works

[IvoLeist]

Dear @skim245, @FADHLyemen and cellxgene_vip developer,

I am a PhD student at CNAG-CRG evaluating right now the potential use of cellxgene/cellxgene_VIP for 3TR (https://www.3tr-imi.eu/)

Since I am also struggling with setting up a functional Dockerfile I thought it could be beneficial to share what I tried so far in your issue. Maybe we could help each other.

Best regards,
Ivo

---

Best shot so far:

#Command Line Interface in cellxgene_VIP is not working
#rpy2 runs into an error (reproduce: import rpy2.robjects as ro)
FROM rocker/tidyverse:4.0.3

RUN apt-get update && apt-get -y install --no-install-recommends \
    git \
    #dependency of igraph \
    libglpk40 \
    #dependecy of Cairo \
    libxt-dev \
    #dependencies for ./config.sh \
    cpio \
    npm 

RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-py38_4.9.2-Linux-x86_64.sh -O ~/miniconda.sh && \
    bash ~/miniconda.sh -b -p /opt/conda && rm ~/miniconda.sh
ENV PATH="/opt/conda/bin:${PATH}"

RUN git clone https://github.com/interactivereport/cellxgene_VIP.git
WORKDIR /cellxgene_VIP

RUN conda update conda && \
    conda config --set channel_priority flexible && \
    conda env update -n base -f VIP.yml

#clone and modify cellxgene
RUN ./config.sh

#faster R package installer
RUN R -e "install.packages('pak',repos='https://r-lib.github.io/p/pak/dev')"

RUN R -e " \
    install_packages_or_die <- function (pkgs,repos='http://cran.us.r-project.org/'){ \
    for (e in pkgs){ \
        pak::pkg_install(e); \
        if ( ! library(e, character.only=TRUE, logical.return=TRUE) ) { \
            stop ('ERROR: failed installing requested package \'',e,'\'') } } }; \
    pkgs <- new.env(hash=T, parent=emptyenv()); \
    install_packages_or_die (c('fgsea','ggpubr','ggrastr','Seurat','viridis'))"

Did not manage to install Seurat:

#Not working because package of conflicts /w Seurat
FROM continuumio/miniconda3:4.9.2

RUN apt-get update && apt-get install -y \
    cpio \
    git \
    xfonts-base \
    npm 

ADD data/* /data/

RUN git clone https://github.com/interactivereport/cellxgene_VIP.git

WORKDIR /cellxgene_VIP

RUN conda update conda && \
    conda config --set channel_priority flexible && \
    conda env update -n base -f VIP_conda_R.yml

ENV LIBARROW_MINIMAL=false

#needs to be installed with conda because r is installed via conda
RUN conda install -c conda-forge \
    r-cairo=1.5-10 \
    r-ragg=0.3.1
    #r-seurat #creates package conflicts with devtools

RUN ./config.sh

RUN R -e 'if(!require(devtools)) install.packages("devtools",repos = "http://cran.us.r-project.org")'

RUN R -e " \
    install_packages_or_die <- function (pkgs, repos='http://cran.us.r-project.org/'){ \
    for (e in ls(pkgs)){ \
        devtools::install_version(e,version=pkgs[[e]],repos=repos); \
        if ( ! library(e, character.only=TRUE, logical.return=TRUE) ) { \
            stop ('ERROR: failed installing requested package \'',e,'\'') } } }; \
    pkgs <- new.env(hash=T, parent=emptyenv()); \
    assign('foreign','0.8-76', pkgs); \
    assign('ggpubr','0.3.0', pkgs); \
    assign('ggrastr','0.1.9', pkgs); \
    assign('arrow','2.0.0', pkgs); \ 
    assign('rmarkdown','2.5', pkgs); \
    assign('tidyverse','1.3.0', pkgs); \
    assign('viridis','0.5.1', pkgs); \
    assign('BiocManager','1.30.10', pkgs); \ 
    install_packages_or_die (pkgs)" 

RUN R -e 'if(!require(fgsea)) BiocManager::install("fgsea")'

#RUN R -e 'if(!require(Seurat)) devtools::install_version("Seurat",version="3.2.3",repos = "http://cran.us.r-project.org")'
#package conflicts

Seurat working but this time I did not manage to install fgsea:

#Not working because of package conflicts /w fgsea
#FROM continuumio/anaconda3:2021.04
FROM continuumio/miniconda3:4.9.2

RUN apt-get update && apt-get install -y \
    cpio \
    git \
    npm 

ADD data/* /data/

RUN git clone https://github.com/interactivereport/cellxgene_VIP.git

WORKDIR /cellxgene_VIP

RUN conda update conda && \
    conda config --set channel_priority flexible && \
    conda env update -n base -f VIP_conda_R.yml

#overwrite r-base (3.6.3/4.0.4/4.0.5 does not work -> package conflicts) 
#packages needs to be installed with conda because R is installed via conda
RUN conda install -c conda-forge \
    r-base=4.0.2 \
    r-cairo=1.5_12.2 \ 
    r-ragg=0.3.1 \
    r-ggrastr=0.2.3 \
    r-arrow=2.0.0 \
    r-biocmanager=1.30.15 \
    r-foreign=0.8_80 \
    r-ggpubr=0.4.0 \
    r-rmarkdown=2.8 \
    r-seurat=3.2.2 \
    r-tidyverse=1.3.0 \
    r-viridis=0.6.1

#Trying to install fgsea
#RUN conda install -c bioconda bioconductor-fgsea
#Error multiple package conflicts

#RUN R -e 'if(!require(fgsea)) BiocManager::install("fgsea")'
#Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) : 
#  namespace ‘ellipsis’ 0.3.1 is already loaded, but >= 0.3.2 is required

#modifies cellxgene 
RUN ./config.sh

[skim245]

@IvoLeist thank you for sharing this.
I've been trying to deploy cellxgene_VIP with Heroku using below codes

git add heroku.yml
git commit -m "Add heroku.yml"
git push heroku master

and getting an error code below

remote: =!= There were problems parsing your heroku.yml. We've detected the following issues:
remote:
remote: build.docker.web references a Dockerfile that does not exist

any thoughts or suggestion?

Thanks,

[IvoLeist]

@skim245
without seeing your heroku.yml and knowing your folder structure I cannot help you with that.

But my random guess is that you have specified the wrong path inside build.docker.web
If your heroku.yml is in the root directory and the Dockerfile is e.g. inside a folder named app:

build:
  docker:
    web: app/Dockerfile

If the heroku.yml and the Dockerfile are in the same folder:

build:
  docker:
    web: Dockerfile

[skim245]

@IvoLeist
here is my heroku.yml

build:
docker:
web: Dockerfile

run:
web:
command:
- cellxgene launch --host localhost --port 5050 --disable-annotations --verbose 042221_scRNA_data.h5ad
and both of my heroku.yml and the Dockerfile are in the same folder.
and do you know which extension to use for Dockerfile ?

[michaeleekk]

I saw the discussion about Dockerfile in this issue, so I'm sharing my Dockerfile here too.

Half of the file is copied from https://github.com/jupyter/docker-stacks/blob/master/base-notebook/Dockerfile and half of the file is from @IvoLeist. I failed to use conda to solve the R dependencies directly, I got the same errors written in @IvoLeist 's Dockerfile. Instead, I followed the Jupyter Notebook Dockerfile and used Mamba to solve the dependencies. It worked pretty well.

I didn't try deploying to Heroku but I managed to deploy on Google Cloud Run. I believe they work in the same way.

FROM debian:bullseye

# ref: https://github.com/jupyter/docker-stacks/blob/master/base-notebook/Dockerfile

# Install all OS dependencies for notebook server that starts but lacks all
# features (e.g., download as all possible file formats)
# - tini is installed as a helpful container entrypoint that reaps zombie
#   processes and such of the actual executable we want to start, see
#   https://github.com/krallin/tini#why-tini for details.
# - apt-get upgrade is run to patch known vulnerabilities in apt-get packages as
#   the ubuntu base image is rebuilt too seldom sometimes (less than once a month)
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update --yes && \
    apt-get upgrade --yes && \
    apt-get install --yes --no-install-recommends \
    git cpio build-essential python3 python3-pip cmake libhdf5-dev cmake llvm npm nodejs jq \
    tini wget ca-certificates sudo locales fonts-liberation && \
    apt-get clean && rm -rf /var/lib/apt/lists/* && \
    echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
    locale-gen

# Configure environment
ENV CONDA_DIR=/opt/conda \
    SHELL=/bin/bash \
    LC_ALL=en_US.UTF-8 \
    LANG=en_US.UTF-8 \
    LANGUAGE=en_US.UTF-8
ENV PATH="${CONDA_DIR}/bin:${PATH}"

# Install conda as jovyan and check the sha256 sum provided on the download site
WORKDIR /tmp

ARG PYTHON_VERSION="3.8.12"

# CONDA_MIRROR is a mirror prefix to speed up downloading
# For example, people from mainland China could set it as
# https://mirrors.tuna.tsinghua.edu.cn/github-release/conda-forge/miniforge/LatestRelease
ARG CONDA_MIRROR=https://github.com/conda-forge/miniforge/releases/latest/download

# ---- Miniforge installer ----
# Check https://github.com/conda-forge/miniforge/releases
# Package Manager and Python implementation to use (https://github.com/conda-forge/miniforge)
# We're using Mambaforge installer, possible options:
# - conda only: either Miniforge3 to use Python or Miniforge-pypy3 to use PyPy
# - conda + mamba: either Mambaforge to use Python or Mambaforge-pypy3 to use PyPy
# Installation: conda, mamba, pip
RUN set -x && \
    # Miniforge installer
    miniforge_arch=$(uname -m) && \
    miniforge_installer="Mambaforge-Linux-${miniforge_arch}.sh" && \
    wget --quiet "${CONDA_MIRROR}/${miniforge_installer}" && \
    /bin/bash "${miniforge_installer}" -f -b -p "${CONDA_DIR}" && \
    rm "${miniforge_installer}" && \
    # Conda configuration see https://conda.io/projects/conda/en/latest/configuration.html
    conda config --system --set auto_update_conda false && \
    conda config --system --set show_channel_urls true && \
    if [ ${PYTHON_VERSION} != "default" ]; then mamba install --quiet --yes python="${PYTHON_VERSION}"; fi && \
    mamba list python | grep '^python ' | tr -s ' ' | cut -d ' ' -f 1,2 >> "${CONDA_DIR}/conda-meta/pinned" && \
    # Using conda to update all packages: https://github.com/mamba-org/mamba/issues/1092
    conda update --all --quiet --yes && \
    conda clean --all -f -y && \
    rm -rf "/home/${NB_USER}/.cache/yarn"


# install R dependencies
RUN mamba install --quiet --yes \
    'r-base' \
    'r-cairo' \
    'r-ragg' \
    'r-ggrastr' \
    'r-arrow' \
    'r-biocmanager' \
    'r-foreign' \
    'r-ggpubr' \
    'r-rmarkdown' \
    'r-seurat' \
    'r-tidyverse' \
    'r-viridis' \
    'r-ellipsis' \
    'r-processx' && \
    mamba clean --all -f -y

WORKDIR /

RUN git clone https://github.com/interactivereport/cellxgene_VIP.git

WORKDIR /cellxgene_VIP

# install Python depdendencies
ADD vip_requirements.txt /cellxgene_VIP/vip_requirements.txt
RUN pip install -r /cellxgene_VIP/vip_requirements.txt

RUN bash config.sh

# install cellxgene Python dependencies if any
RUN pip install -r /cellxgene_VIP/cellxgene/server/requirements.txt

The following is the vip_requirements.txt used in the Dockerfile, extracted from the VIP.yml in this repo.

aniso8601==8.1.1
anndata==0.7.4
attrs==20.3.0
batchglm==0.7.4
bleach==3.2.3
boto3==1.14.39
botocore==1.17.44
brotli==1.0.9
cffi==1.14.4
chardet==4.0.0
click==7.1.2
cycler==0.10.0
dask==2021.1.1
defusedxml==0.6.0
diffxpy==0.7.4
docutils==0.15.2
entrypoints==0.3
fastobo==0.9.3
flask==1.1.2
flask-compress==1.8.0
flask-cors==3.0.10
flask-restful==0.3.8
flask-server-timing==0.1.2
flask-talisman==0.7.0
flatbuffers==1.12
flatten-dict==0.3.0
fsspec==0.7.4
get-version==2.1
gunicorn==20.0.4
h5py==2.10.0
idna==2.10
iniconfig==1.1.1
itsdangerous==1.1.0
jinja2==2.11.2
jmespath==0.10.0
joblib==1.0.0
jsonschema==3.2.0
jupytext==1.6.0
kiwisolver==1.3.1
legacy-api-wrap==1.2
llvmlite==0.35.0
markdown-it-py==0.5.8
markupsafe==1.1.1
matplotlib==3.3.4
mistune==0.8.4
natsort==7.1.1
nbconvert==5.6.1
nbformat==5.1.2
networkx==2.5
numba==0.52.0
numexpr==2.7.2
numpy==1.19.5
packaging==20.9
pandas==1.2.1
pandocfilters==1.4.3
pathlib2==2.3.5
patsy==0.5.1
pillow==8.1.0
plotly==4.8.1
pluggy==0.13.1
py==1.10.0
pyarrow==1.0.1
pycparser==2.20
pyparsing==2.4.7
pyrsistent==0.17.3
pytest==6.2.2
pytz==2020.5
pyyaml==5.4.1
requests==2.25.1
retrying==1.3.3
rpy2==3.3.5
s3fs==0.4.2
s3transfer==0.3.4
scanpy==1.6.1
scikit-learn==0.24.1
scipy==1.6.0
seaborn==0.11.1
setuptools-scm==5.0.1
sinfo==0.3.1
sparse==0.11.2
sqlalchemy==1.3.22
statsmodels==0.12.1
stdlib-list==0.8.0
tables==3.6.1
testpath==0.4.4
threadpoolctl==2.1.0
tiledb==0.7.7
toml==0.10.2
toolz==0.11.1
tqdm==4.56.0
tzlocal==2.1
umap-learn==0.4.6
urllib3==1.25.11
webencodings==0.5.1
werkzeug==1.0.1