A secure sandbox for running and judging code generated by LLMs.

Please view the complete documentation at https://bytedance.github.io/SandboxFusion/ .

This fork adds execution modes for running Codewars kata test suites. Each mode sets up the language-specific test framework, dependencies, and module structure that Codewars katas expect, so that the original test code runs unchanged.

The sandbox runs as a Docker container and exposes a single /run_code endpoint. The validation pipeline sends each (kata, language) pair as a POST request:

curl -X POST http://localhost:8082/run_code \
  -H "Content-Type: application/json" \
  -d '{
    "code": "<test_code>",
    "language": "codewars_python",
    "files": {"solution.py": "<base64_encoded_solution>"},
    "run_timeout": 32
  }'

The code field contains the kata's test suite, language selects the execution mode (codewars_python, codewars_jest, etc.), and files provides the candidate solution as a base64-encoded file. The response includes compile_result and run_result, each with status, return_code, stdout, and stderr. A pair passes validation when the run finishes with return code 0.

• Python — copies the codewars_test framework into a temp dir, imports the solution via from solution import *, and runs the test suite.
• JavaScript — prepends the solution to the test code, extracts global variable declarations to avoid const/let redeclaration errors, and runs via Mocha with @codewars/test-compat providing the legacy Test.* API.
• Java — renames the solution file to match its public class name and runs JUnit with the appropriate classpath.
• Go — creates a codewarrior/kata/ module structure and handles both package kata_test and package kata.
• Rust — concatenates solution + tests into src/lib.rs and runs cargo test with pre-compiled dependencies.
• C++ — includes Igloo/Snowhouse headers, auto-adds main() if absent, and compiles with clang++.

docker build -f ./scripts/Dockerfile.codewars -t code_sandbox:codewars .
docker run -d --rm --privileged -p 8082:8080 code_sandbox:codewars make run-online

• Features
• Cite
• Contributing
• License

Code Runner: Run and return the result of a code snippet

Supported languages:

• Python (python, pytest)
• C++
• C#
• Go (go, go test)
• Java (javac, junit)
• NodeJS
• Typescript (tsx, jest)
• Scala
• Kotlin
• PHP
• Rust
• Bash
• Lua
• R
• Perl
• D
• Ruby
• Julia
• Verilog
• CUDA (GPU)
• Python (GPU)

Jupyter mode kernels:

• python3

Online Judge: Implementation of Evaluation & RL datasets that requires code running

• HumanEval
• MultiPL-E HumanEval
• Shadow Humaneval
• CodeContests
• MBPP
• MBXP
• MHPP
• CRUXEval
• NaturalCodeBench
• PAL-Math
• verilog-eval

@misc{bytedanceseedfoundationcodeteam2025fullstackbenchevaluatingllms,
      title={FullStack Bench: Evaluating LLMs as Full Stack Coders}, 
      author={Bytedance-Seed-Foundation-Code-Team and : and Yao Cheng and Jianfeng Chen and Jie Chen and Li Chen and Liyu Chen and Wentao Chen and Zhengyu Chen and Shijie Geng and Aoyan Li and Bo Li and Bowen Li and Linyi Li and Boyi Liu and Jiaheng Liu and Kaibo Liu and Qi Liu and Shukai Liu and Siyao Liu and Tianyi Liu and Tingkai Liu and Yongfei Liu and Rui Long and Jing Mai and Guanghan Ning and Z. Y. Peng and Kai Shen and Jiahao Su and Jing Su and Tao Sun and Yifan Sun and Yunzhe Tao and Guoyin Wang and Siwei Wang and Xuwu Wang and Yite Wang and Zihan Wang and Jinxiang Xia and Liang Xiang and Xia Xiao and Yongsheng Xiao and Chenguang Xi and Shulin Xin and Jingjing Xu and Shikun Xu and Hongxia Yang and Jack Yang and Yingxiang Yang and Jianbo Yuan and Jun Zhang and Yufeng Zhang and Yuyu Zhang and Shen Zheng and He Zhu and Ming Zhu},
      year={2025},
      eprint={2412.00535},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2412.00535}, 
}

Docker

Build the image locally:

docker build -f ./scripts/Dockerfile.base -t code_sandbox:base .
# change the base image in Dockerfile.server
sed -i '1s/.*/FROM code_sandbox:base/' ./scripts/Dockerfile.server
docker build -f ./scripts/Dockerfile.server -t code_sandbox:server .
docker run -d --rm --privileged -p 8080:8080 code_sandbox:server make run-online

Manual

Prerequisites: conda, poetry

To install the sandbox service:

conda create -n sandbox -y python=3.12
conda activate sandbox
poetry install
# to build the real docs, run `cd docs && npm ci && npm run build`
mkdir -p docs/build
make run-online

Please refer to scripts/Dockerfile.base for the runtime of each supported language, and scripts/Dockerfile.server for the installation of extra packages for python and nodejs.

For example, if you want to support execution of python code, you should install install-python-runtime.sh

cd runtime/python
bash install-python-runtime.sh

Refer to installation section for the setup of development environment.

Run all unit tests:

make test

Run a specific unit test (allows you to see stdout):

make test-case CASE=test_java_assert

Run a specific unit test with pdb:

make test-case-pdb CASE=test_java_assert

Format the code:

make format

Copyright 2024 Bytedance Ltd. and/or its affiliates

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

sine 💻 🔣 🎨

Qi Liu 💻 🔣

Shulin.Xin 💻 🔣

可爱猫猫是梓喵 💻 🔣

Xingwen Li 🎨

aoyanli 💻 🔣

Magic Mai 💻 🔣