custom data type can't be created in Appian community edition ( Free version )

Yes that must have been disabled due to security concern. But you can do everything what a CDT can do with recordType. Just try a bit.

can you collaborate more? 
I want to use the custom data type in the expression rules like comparing 2 field values and so on. 

Where ever it has mentioned to define rule input of type CDT define that rule input as type of the record. 

example if lesson was asking you to create a rule input of type employee cdt which is referring to employee table then create the rule input of type employee record 

so based on my understanding, record type is like the full table, while the CDT is the customized record type where it has certain fields from the original Record type? 

Not exactly. CDTs are more like getter setter class from traditional programming world where as record type  reffer to record objects in Appian.

Both CDT and records refer to tables and each column from tables can be refferenced easily using fields in CDT and records.

In  my experience once you start creating rule inputs of type record types you would not like to create CDTs and rule input of type CDTs

Not exactly. CDTs are more like getter setter class from traditional programming world where as record type  reffer to record objects in Appian.

Both CDT and records refer to tables and each column from tables can be refferenced easily using fields in CDT and records.

In  my experience once you start creating rule inputs of type record types you would not like to create CDTs and rule input of type CDTs