This problem generally start when you are having an existing custom workflow and there are instances of workflow running. You do some changes to the workflow and reinstall the assembly in the GAC. The new assembly is basically overwriting the existing one (it has same signature). When you try to complete the existing workflow instances after updating the assembly, you may get "Task is locked by running workflow instance" error or it just hangs in "In Progress" state.
So what did happen in background?
The public interface of the workflow, which has been serialized in the database, have changed and now the new assembly does not know how to de-serialize the workflow data from the database. At this point the workflow will fail as it is unable to serialize the current state and looks like it hanged in "In Progress" state.
Then how should I upgrade the assembly for a workflow?
Upgrading running instances of a workflow is a tricky business because the workflow doesn't know, if it's current state is, before or after a change in the assembly. When anything changes in public interface of workflow assembly, serialization/de-serialization of workflow data fails. The new assembly of workflow does not know how to load the older serialized data.
If you are just changing the code logic within the methods, you should be able to update the workflow assembly without breaking the existing workflow instances. But if you add/remove/modify any activity or add/remove any properties/methods from the code and update the existing workflow assembly, that would break the existing instances.
In nutshell, any kind of changes to the interface (public code interface or workflow design interface, which is serialized) will break the workflow.
The suggested way to upgrade the workflow is to create a new version of workflow and install it in GAC and SharePoint. Mark the existing workflow to "No New Instance", which will not create any new instances of old workflow.
Use the new workflow assembly for new instances, and the old workflow assembly for old instances. To do this, you need to update the assembly version to the next one (e.g. 2.0 from 1.0) and adjust the workflow.xml file to point to 2.0, then keep both 1.0 and 2.0 in the GAC. When you reinstall, new instances will use that new version, and old instances will continue to use the old and when all the older workflow instances have finished, you can remove the 1.0 version from the server.