Skip to content

Feature Server image won't start in an OpenShift cluster #4095

@tchughesiv

Description

@tchughesiv

Expected Behavior

The python Feature Server image should run in an OpenShift cluster without issue.

Current Behavior

The container doesn't start successfully, throws this error -

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/.cache/fissix/21.11.13'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/.cache/fissix'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/feast", line 5, in <module>
    from feast.cli import cli
  File "/usr/local/lib/python3.8/site-packages/feast/cli.py", line 44, in <module>
    from feast.repo_upgrade import RepoUpgrader
  File "/usr/local/lib/python3.8/site-packages/feast/repo_upgrade.py", line 5, in <module>
    from bowler import Query
  File "/usr/local/lib/python3.8/site-packages/bowler/__init__.py", line 13, in <module>
    from .imr import FunctionArgument, FunctionSpec
  File "/usr/local/lib/python3.8/site-packages/bowler/imr.py", line 12, in <module>
    from fissix.fixer_util import LParen, Name
  File "/usr/local/lib/python3.8/site-packages/fissix/fixer_util.py", line 7, in <module>
    from .pygram import python_symbols as syms
  File "/usr/local/lib/python3.8/site-packages/fissix/pygram.py", line 30, in <module>
    python_grammar = driver.load_packaged_grammar("fissix", _GRAMMAR_FILE)
  File "/usr/local/lib/python3.8/site-packages/fissix/pgen2/driver.py", line 153, in load_packaged_grammar
    return load_grammar(grammar_source)
  File "/usr/local/lib/python3.8/site-packages/fissix/__init__.py", line 36, in load_grammar
    gp = _generate_pickle_name(gt) if gp is None else gp
  File "/usr/local/lib/python3.8/site-packages/fissix/__init__.py", line 28, in _generate_pickle_name
    CACHE_DIR.mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.8/pathlib.py", line 1292, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.8/pathlib.py", line 1292, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/.cache'

Steps to reproduce

Against an OpenShift cluster -

$ helm install feast-feature-server feast-charts/feast-feature-server --set feature_store_yaml_base64=$(base64 -i feature_store.yaml)

Specifications

  • Version: 0.35.0
  • Platform: OpenShift

Possible Solution

The issue is that the OpenShift restricted SCC, which is configured against namespaces by default, forces containers to run with a random uid, and a root gid. The solution is to pre-create the /.cache dir with the proper permissions during the image build.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions