Upload and analyze a file in a User Input Task

You cannot access temporary file data before submission, implement a two-step validation process where the first User Input Task allows file upload and submission using a!submitUploadedFiles() to save the file to the system. After submission, use Excel Tools plugin functions like readexcelsheetpaging() in your process model to count the rows, then route to a second User Input Task that either displays a validation error message if rows exceed 500 or allows the user to continue with the process if the file meets the criteria.
Make user feel like process not 2 steps.

With the latest version of Appian, you have direct access to a file without submit.

Whether you are using folder stored documents or record managed ones you can read the data in the saveInto of the fileUpload component and using that validate the data. Here is a code you can refer which uses readexcelsheetpaging() which comes as part of Excel Tools plugin. 

a!localVariables(
  local!dataLength,
  {
  a!fileUploadField(
    label: "Docs",
    target: cons!HSS_DOCUMENTS_TARGET,
    maxSelections: 1,
    saveInto: {ri!doc,
    a!save(
      local!dataLength,
      readexcelsheetpaging(
       ri!doc,
        0,
        a!pagingInfo(1, 50)
      ).totalCount
    )},
    instructions: ri!doc,
    validations: if(local!dataLength>20,"Maximum 20 rows are allowed",""),
    value: ri!doc
  )})

P.S My code is written & tested in Appian v25.2 & v25.3. Check if it works for you depending on the version you are working on! Let me know how it goes.