Here is a use case There is a workflow which goes this way, the recor

Does this user who initiated the workflow need to be able to still see the tasks under his "Task" tab?

I'm curious because if you just assign it to all members of Group C or Group D and exclude that one user then they will not get the task assigned which means they can't pick up the task. If your record UI shows the progress of the record through the workflow they should still be able to follow what is happening to the data/workflow.

To exclude a user from the picked assignees for a task you could use a rule like:
difference(
getdistinctusers(ri!tempGroup),
loggedinuser()
)

Replace 'loggedinuser()' with your initiator (or rule input) and ri!tempGroup with another group.

Hopefully this is in line with what you mean.

@vinayakb You have said that the record flows between Group A -> Group B -> Group C -> Group D. I guess it would be better to see as mentioned below and that's how the assignment of user input tasks should be controlled as per my opinion:

Group A => Group B minus (the Process Initiator/Task Owners from processes that belongs to Group A) => Group C => Group D minus (the Process Initiator/Task Owners from processes that belongs to Group A)


When it comes to assignments that should go to Group B/Group D, the task assignees should be the intended assignees minus the people who already participated in tasks of Group A. These task assignees can be controlled in some of the following ways:
1. Create Dynamic Groups. Let's take the example of Group - B. The Dynamic Group should initially consist of members from the Group - B. Post completion of tasks from Group - A, remove those members from the Dynamic Group, who actively participated in the tasks or processes of Group - A. Use the same group as task assignees of the assignments that needs to be performed by Group B when you come across a situation where you need to avoid participants of Group - A tasks. Off-course we store the fact in the database at the same time, but Dynamic Groups will be used in managing Task Assignments primarily.
2. Store the fact in the database that a set of user(s) has participated in the tasks of Group - A. Let's take the example of Group - B. While assigning the tasks at Group B, instead of using the Group B group object, make use of User or Group Array Process Variable and populate the PV with all the members of Group B excluding those users that are present in the database as active participants of Group - A tasks.

Dynamic Groups (and Database):
Advantages with the Dynamic Groups is that, they can be managed (membership updates) easily and utilised (controlling security of Appian objects such as Related Action, Views etc, hiding the contents on Dashboard) effectively.
Down side is that, creation of dynamic groups is directly proportional to volume of instances and so this may cause a issue further as many opine that having those many groups can be a issue(for instance, I have heard of a best practice stating that a user can't be member of more than 100 groups).
We need to bear in mind that upon completion of the Work Object, we need to delete the associated dynamic groups, and all the artifacts related to the Work Object should now start looking at database for Group details. Also care should be taken that Dynamic Group and the data in the Database are in sync.

Database:
Task Management would be slightly difficult here. For instance, let's say a task has been assigned to a set of users whose assignees are sourced from Database information. In order to update the assignees, it's not as easy as we do for a task whose assignees are sourced from a Group Object. Also at times, we need to make a query to the database which would have been completed with a simple People function in case of Groups.

The most elegant solution is to have a new feature in the product that implement "Two-Man Rule" aka "Four-eye Principle" by accepting list of groups or users to be excluded from the task assignment.

en.wikipedia.org/.../Two-man_rule

Without that, any solutions would come with a trade-off.

Thanks All! Interesting answer by Rawich, will take a look :-)