UnrealCV is an open source project to help computer vision researchers build virtual worlds using Unreal Engine (UE). It extends UE4/5 with a plugin by providing:

1. A set of UnrealCV commands to interact with the virtual world.
2. Communication between UE4/5 and an external program, such as Pytorch.

UnrealCV can be used in two ways. The first one is using a compiled game binary with UnrealCV embedded. This is as simple as running a game, no knowledge of Unreal Engine is required. The second is installing UnrealCV plugin to Unreal Engine (UE4/5) and use the editor of UE to build a new virtual world.

Please read Tutorial: Getting Started to learn using UnrealCV.

Images generated from the technical demo RealisticRendering

News

• 2024.06 👏 UnrealCV Client has updated to 1.1.0 with easy-to-use Python APIs and launcher! You can pip install the latest version.
• 2024.06 👏 UnrealCV has supported UE 5.4! You can download the 5.2 branch and play with it. Thanks to russkel for the contributions.
• 2024.03 👏 UnrealCV has supported UE 5.2! You can download the 5.2 branch and play with it. Thanks to mehmetcanakbay for the contributions.
• 2023.05 👏 We release a paper list to show how UnrealCV is used in research. Welcome to add your paper to this list!

Tutorials

• Getting Started - The basics of using UnrealCV, start from here.

• Generate an image dataset with ground truth - An iPython notebook showing how to generate an image dataset with rich ground truth from a virtual scene.

• Integrate with Faster-RCNN - Show how to use images from a video game for testing faster-RCNN.

• Install Plugin to UE4/5 - How to install UnrealCV plugin into UE4/5.

• Integrate with OpenAI Gym - How to combine UnrealCV and OpenAI Gym interface for reinforcement learning.

Reference

• Architecture - Technical Details of UnrealCV

• Command List - All available commands provided in UnrealCV

• Model Zoo - A list of compiled games with UnrealCV integrated

• Demos - A demo video about applications of UnrealCV.

• Paper List - A curated list of papers using UnrealCV.

Citation

If you found this project useful, please consider citing our paper

@article{qiu2017unrealcv,
  Author = {Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao,Zihao Xiao, Tae Soo Kim, Yizhou Wang, Alan Yuille},
  Journal = {ACM Multimedia Open Source Software Competition},
  Title = {UnrealCV: Virtual Worlds for Computer Vision},
  Year = {2017}
}

Contact

UnrealCV is an open-source project created by students from Johns Hopkins University and Peking University under the supervision of Prof. Alan Yuille and Prof. Yizhou Wang. The team thanks NSF Expedition in Computing, Visual Cortex On Silicon and Deep Intermodal Video Analytics (DIVA) for supporting this work. It is maintained by the UnrealCV team, see the team member list.

If you have any suggestion or interested in using UnrealCV, please contact us.