In the context of an on-premise server, we have the capability to manage log files by archiving, compressing, and uploading them to S3 for efficient on-premise server disk storage management. However, when our application runs on Appian Cloud, we are aware that we can access the log files via the /logs URL, as shown in the attached screenshot. While it's possible to manually download these files one by one, compress them, and then upload them to S3, we are trying to find the best and most efficient approach to streamline this process. Any insights on the quickest and most intelligent way to handle this step would be greatly appreciated please.
Discussion posts and replies are publicly visible
From my perspective, not having to care about things like this, is one of the main reasons to use the cloud. If you do not have any very specific use case or requirement, I would just leave it as it is.
If you are in the cloud, you shouldn't have to worry about disk space (aside from the cost) but you don't have much control on how those log files are managed.
If you are looking to put those logs somewhere for further analysis or ingest them in a SIEM (i.e. DataDog, Splunk etc.) - I would suggest you take a look at Log Streaming for Appian Cloud - https://docs.appian.com/suite/help/23.3/Log_Streaming_for_Appian_Cloud.html.
Thanks Steve / Mathieu,
Cost optimization represents a fundamental aspect of well-architected standards in the realm of cloud applications. In pursuit of storage and cost optimization, some may prefer removing archived logs after a 30-day period. As we aware, this uploading into S3 bucket approach proves more cost-effective compared to the alternative of expanding disk storage solely to accommodate archived log files within the Appian Cloud infrastructure.
In on-premises environment, we have the flexibility to upheld a data retention policy of 12 months for logs, primarily for historical data tracking and debugging purposes. This could be nice to have feature, I'm exploring options within the Appian App Market or alternative solutions to facilitate bulk downloading of Appian Cloud log files. Your provided link is greatly appreciated, and I will further investigate this resource. Thanks again.