I am brand new to Julia, and this is my first project, writing software for an archiving/backup system, from a multitude of different sources. I am trying to read in and processes files (often up to a gigabyte big) of what is essentially records of text data (one record per line) but the meaning and…

At the risk of a certain person falsely claiming that I’m “using every thread as a chance to plug your own work”, I think that you might be interested in the packages in the JuliaString org, in particular Strs.jl . Unfortunately, it is still in the process of being registered, so you might want to j…

[image] Yogi3: For most processing I simply do not need to discover whether if is Unicode, symbols from a ‘code-page’, peculiar to the owner of that record, a corruption, parity or whatever). If you neither know nor care what the encoding is, why do you need a string type at all? Why not jus…

He did say “For most processing”, not for all processing, and he still may wish to search the strings for ASCII values, etc, without worrying about whether or not bytes with the high bit set represent part of a UTF-8 sequence, a S-JIS or EUC sequence, an ISO character or some Windows CP character. …

Are you using 0.6? In the latest Julia versions (master / soon-to-be-released 0.7-alpha), the standard String type can hold arbitrary data, but gives it a UTF-8 like interpretation and display. On 0.6 and earlier some operations insisted on valid UTF-8 data in Strings, which indeed did lead to annoy…

Yet another alternative to my statement 'Alternatively I can convert into an array of characters each of which is 32bits (4 bytes) long, and then convert that back into to a byte string again. The actual processing in this latter form seems much harder. (To set about my problem regarding the whole …

Why is it that that whenever I ask a question the simple solution will always be available – err – sometime in the near future? Yes if Pkg.add(“Strs”) is as simple as you indicate, without experiences mentioned in my question, that is great when it comes. I look forward to this. I would love to ge…

Pkg.clone("https://github.com/JuliaString/Strs.jl")

| | || | | | (| | | Version 0.6.2 (2017-12-13 18:08 UTC) / |_‘|||_’_| | Official http://julialang.org/ release |__/ | x86_64-pc-linux-gnu julia> Pkg.clone(“ https://github.com/JuliaString/Strs.jl ”) INFO: Initializing package repository /home/richard/.julia/v0.6 INFO: Cloni…

I personally found it extremely hard to parse that large block of text, but from what I understand, you have a file consisting of lines of the form HDP\n where the H is the fixed-length sha1 hash, D is a fixed-length date, and P is an arbitrary length path, and \n is the newline (\x0a) character. …

Assuming that my interpretation is correct (and the last separator is always /), then something like the following should work: function append_5chars(newfile, oldfile) open(newfile,"a") do f_out open(oldfile) do f_in while !eof(f_in) write(f_out, read(f_in, …

Problem processing non utf8 string

New to Julia

Topic		Replies	Views
Julia's UTF-8 handling [vs. new Python's 3.7 UTF-8 PEP 540] Internals & Design	29	5205	January 24, 2018
Problems with deprecations of islower, lowercase, isupper, uppercase Internals & Design	179	15850	January 1, 2018
Is Julia well-suited for string manipulation? General Usage strings	24	3444	March 24, 2023
String indices : byte indexing feels wrong New to Julia strings , unicode	18	1805	December 5, 2023
Julia simple String process and file I/O is slow Performance performance , parsing	12	3157	December 23, 2020

Problem processing non utf8 string

Related topics