i have a sub process which is called by the main process form different activiti

When configuring your sub-process, check " Do not create if active instances exist. " in the Other tab under "When this node is re-executed:"

Under the 'other'tab of the subprocess, there is a checkbox that says do not create instance if an instance already exist. This is one option but this wont guarentee duplicate as it will start another instance if the previous one completes.
I'm not sure what the use case here is.. but you can use a check before you initiate the subprocess and skip if an instance has started. Use a simple pv! to set that it has started.

configured but its still creating instances of sub process

As Sathya suggested, an XOR gateway before the sub-process that checks for a pv! being true(), then sets the pv! to false and executes the sub-process node should suffice. Alternatively, you could also setup and Exception event on the sub-process.