With increasing usage of smart devices such as mobile phones and wearable devices, the modern society has progressed one step closer to the coming age of ubiquitous computing. However modern smart devices are limited by their computing powers and hardware technologies, hindering its achievement of seamless interaction with other physical objects and electronic devices and thorough interpretation of physical environment. Immersive Interactive Multimedia System attempts to overcome the technological drawbacks through use of location/context aware sensors, thereby conveying the physical information of the environment in to the cyber space. By utilizing different sensors and signal processing techniques available for modern smart phone, the System could achieve intuitive and simple gesture interaction between the user and multiple objects. By integrating this interactive System with multimedia contents, one will be able to attain high engagement level from the audience and present them an immersive user experience.