How to create a log file storage

I have a lot of log files, each perhaps 100 MB of size, compressed a lot less, but they also can be bigger.

I would like to store them at a central location together with some meta data. It would be nice if the meta data is version controlled.

Each user (like 20 people) should be able to upload and download the log files and to analyse them, using a Julia script.

What kind of storage should I use? We have azure in the company, but I also thought about git-lfs for example.

Any suggestions?

There is a lot of options that mostly depend on cost performance ratio and technical setup requirements. I would consider the basics: S3 compatible storage and NFS share. The first natively supports version control. Not sure about the second. However, in both cases you can setup your own version control system. As for the S3 and Julia, you may find the following info useful. How to access S3 compatible CEPH bucket?. Of course Git Large File Storage is another option but 100MB is not that large IMHO. I would like to add that quite a lot of cloud providers offer products designed especially for logging and log access. I am aware about the one at IBM Cloud, do not have much experience with Azure.

1 Like