AI tools to write (Julia) code (best/worse experience), e.g. ChatGPT, GPT 3.5

Offtopic
10

[I realize this is a rather long (too long?) comment, and maybe incomplete, but I found it in drafts, and just posted it so I could write and post the comment after it… I may edit this one later, or not…]

My own experience/chat log below, first answering. Note, intriguingly it knows about Genie.jl, and seemed to know how to it (until I checked the code), but weeks later it knew nothing about Genie (or Stipple), when asking differently about it, at first. @essenciary And Stipple.jl is NOT a Julia package for “stippling” (rather a “reactive UI library” for the web), “a technique for creating a halftone image using a set of small dots.” But it could have been, that seems to be a thing, and ChatGPT just made it up, it tries to “please”, or “guess”, does some “pattern matching”. It’s not a word salad, all coherent and plausible. I’m guessing it’s much better for more mainstream languages like Python, just a question of more training data, just as it gave me good answers for e.g. tricky physics questions.

[Exclusive] What [Yann] LeCun Thinks of ChatGPT?

“I don’t think any company out there is significantly ahead of the others,” said [Yann] LeCun, VP and chief AI scientist at Meta AI, in an exclusive interview with Analytics India Magazine, when asked about the growing popularity of OpenAI’s ChatGPT.

LeCun said that a lot of people are, in fact, working on language models with slightly different approaches. He said there are three to four companies producing GPT-X-like models. “But, they [OpenAI] have been able to deploy their systems in such a way that they have a data flywheel. So, the more feedback they have, the more the feedback they get from the system, and later adjust it to produce better outputs,” he explained.

“I do not think those systems in their current form can be fixed to be intelligent in ways that we expect them to be,” said LeCun. He said that data systems are entertaining but not really useful. […]

I wouldn’t expect ChatGPT to be able to do all calculations (no more than humans), especially with irrationals. There’s no one true answer, only in infinite stream of numbers after the decimal point (which I don’t expect it to do, even though a spigot algorithm is available, e.g. for pi, it just wouldn’t be very to end up in such an “infinite” printing loop), and everything else is an approximation, and none clearly better than others.

Large language models (LLMs) are are I believe known to be better for symbolic math, some other AI LLM is already good for some college level symbolic math or word problems, intriguingly more so than than for high school (arithmetic) math. The models don’t yet do thinking, logical steps, though they might be in many cases be very good at fooling you into thinking they do that. There’s already an LLM out there that combines with a programming language (Python as I recall), i.e. spells out its “though process”, what you need to calculate, and then runs it with Python. That should be good enough for a lot of things such as calculating (approximation to) irrationals, or formulas, for problems with or involving. There’s no need to calculate e.g. square root “in your head”, we (your neural network/brain) already does that when we need to, and there’s no need for an artificial neural network to do it either, or a good reason to expect it to be much better at it than the human brain.

ChatGPT seems to be the best AI out there, for well chat, what it’s made for, but I suspect AI will get a whole lot better from now on, when just the already known proven ideas I know of (with code) have been integrated into one AI. E.g. a) I know of a different AI that show its though process and then hands the calculation for you into Python so it would calculate 5th root correctly. b) I don’t believe ChatGPT is optimized in any way for programming (or poems or translation, though I managed that bypassing logic of OpenAI to disable it), like other LLMs, just happens to be trained on programs too as text.

I don’t know for sure if ChatGPT DOESN’T have specific capabilities made for programming since other AI has that already. The point (or one of) LLMs is that the parse tree of human language is ambiguous, and it handles/infers it without it being provided (you don’t have to provide the correct one, often there are many that apply, sometimes all valid as with jokes, the point being to be ambiguous, sometimes some parse trees are clearly absurd and not even noticed by humans).

For programming there’s only one correct parse tree (same with LaTeX formulas, which I know many models, and ChatGPT, do have support for, why e.g. a² + b² = c² makes sense, isn’t the same as a2 + b2 = c2), and some AI models, CodeAI from Microsoft if I recall, handle parse trees and need to have specific code for e.g. Python (don’t have for Julia, last I checked). I believe ChatGPT has no such unambitious parsing logic, but could have, and even if then most likely none yet for Julia (or Genie, see below, with it very obscure). That it works at all, is a minor miracle IMHO.

Large pre-trained language models such as GPT-3, Codex, and Google’s language model are now capable of generating code from natural language specifications of programmer intent. We view these developments with a mixture of optimism and caution. On the optimistic side, such large language models have the potential to improve productivity by providing an automated AI pair programmer for every programmer in the world. On the cautionary side, since these large language models do not understand program semantics, they offer no guarantees about quality of the suggested code. In this paper, we present an approach to augment these large language models with post-processing steps based on program analysis and synthesis techniques, that understand the syntax and semantics of programs. Further, we show that such techniques can make use of user feedback and improve with usage. We present our experiences from building and evaluating such a tool jigsaw, targeted at synthesizing code for using Python Pandas API using multi-modal inputs. Our experience suggests that as these large language models evolve for synthesizing code from intent, jigsaw has an important role to play in improving the accuracy of the systems.
[…]
Variable Name transformations: PTLMs make variable refer-
encing errors (as noted in Section 2) because of its implicit bias
towards common dataframe names such as df, df1, df2, dfout
and also because users tend to not specify variables explicitly in
their queries. We find that this simple post-processing module gives
an improvement of 10%-30% for Codex and 10%-15% for GPT-3

Semantic Repair: We see that the semantic repair post-processing
module improves absolute performance of Codex by ∼ 5% and of
GPT-3 by 6%-11%. This underscores the significance of using pro-
gram analysis techniques to augment language models that do not
have inherent understanding of code semantics.

5.5 Beyond Pandas
To test the generality of Jigsaw, we did a preliminary evaluation
with 25 TensorFlow [6 ] tasks sourced from TF-coder [38 ] and online
forums like StackOverflow.

https://www.microsoft.com/en-us/research/publication/code-mixed-parse-trees-and-how-to-find-them/

I didn’t try any of the code yet (it just seemed valid, and my hours long conversation in text form, e.g. one the hedge fund business, e.g. ELI5ing it, gave impressive answers).

[I ran this in December, then again the same series of question again in 2023, and got similar results, though different [correct JavaScript] updateText() no referring to time, though followup question gave code referring to time… though slightly differently <?php echo date('h:i:s a'); ?>]

: Can you make a simple web page?

[I believe correct code, I would have liked a bit more, such as UTF-8 declaration, but good enough, I believe browsers should default to that anyway if not specified.]

: Can you make it dynamic?

Sure, here is a simple example of a dynamic web page:

[seemingly valid]

In this example, the web page uses JavaScript to display the current time in the <span> element with the id “time”. The updateTime() function is called every second (using the setInterval() function) to update the time display. This simple example shows how JavaScript can be used to add dynamic elements to a web page.

: Can you do the same from the server-side?

Sure, here is a simple example of a dynamic web page that uses server-side scripting:

<!DOCTYPE html>
<html>
  <head>
    <title>My Dynamic Web Page</title>
  </head>
  <body>
    <h1>Welcome to My Dynamic Web Page</h1>
    <p>This is a simple web page that uses server-side scripting to display the current time.</p>
    <p>The time is: <?php echo date("Y-m-d H:i:s"); ?></p>
  </body>
</html>

[…]

: Can you do it with the Julia language?

Yes, it is possible to create dynamic web pages using the Julia programming language. Here is a simple example:

# MyDynamicWebPage.jl

using HTTP

function handle_request(req::Request)
  res = Response(200, "text/html")
  res.body = """
  <!DOCTYPE html>
  <html>
    <head>
      <title>My Dynamic Web Page</title>
    </head>
    <body>
      <h1>Welcome to My Dynamic Web Page</h1>
      <p>This is a simple web page that uses the Julia programming language to display the current time.</p>
      <p>The time is: $(now())</p>
    </body>
  </html>
  """
  return res
end

server = HTTPServer(handle_request)
run(server, 8080)

In this example, the handle_request() function is used to handle incoming HTTP requests. This function creates an HTML response that includes the current time, using the now() function from the Dates package. The web server listens on port 8080 and serves the dynamic web page to any clients that connect. This example shows how the Julia language can be used to create dynamic web pages on the server-side.

[the second time around I got an example with Mux.jl rather than HTTP.jl:

[…] Here is an example of a simple Mux.jl application that generates a dynamic web page:

using Mux

# Define a route that displays the current time
@route("/time") function time_route()
    now = Dates.now()
    text = "The current time is $now"
    return text
end

# Start the Mux.jl server
run_mux()

This code defines a route called /time that returns the current time as a string. When you visit the /time route in a web browser, the server will execute the time_route() function and send the resulting text as the response.

You can also use other Julia web frameworks such as Genie.jl, Flux.jl, or Zygote.jl to create dynamic web pages. You can also use Julia to generate HTML code directly, or to interface with other web development tools and libraries such as Node.js, React, or Angular.

[It’s a bit odd to refer to Flux and Zygote there… while you CAN make a web page and mix those in, they are not meant for web pages specifically. I’m guessing, in some code in the training data some did.]

: Can you use Genie to do it?

Yes, it is possible to use the Genie programming language to create dynamic web pages. Here is a simple example:

# MyDynamicWebPage.gs

import Dates

init
  var server = new Server()
  server.bind("0.0.0.0", 8080)
  server.run()

@route "/"
def handle_request():
  html = """
  <!DOCTYPE html>
  <html>
    <head>
      <title>My Dynamic Web Page</title>
    </head>
    <body>
      <h1>Welcome to My Dynamic Web Page</h1>
      <p>This is a simple web page that uses the Genie programming language to display the current time.</p>
      <p>The time is: $(Dates.now())</p>
    </body>
  </html>
  """
  return html

In this example, the handle_request() function is used to handle incoming HTTP requests. This function creates an HTML response that includes the current time, using the Dates.now() function. The Server object listens on port 8080 and serves the dynamic web page to any clients that connect. This example shows how the Genie language can be used to create dynamic web pages on the server-side.

[When rerunning the question in 2023 “Genie” was “Genie.jl” in this context, but not the Genie programming language.]

Yes, you can use the Genie web framework to create dynamic web pages in Julia. Genie is a lightweight and flexible web framework that is built on top of the HTTP package and the Mux routing system. It provides a simple and intuitive syntax for defining routes, handling requests and responses, and rendering templates.

Here is an example of a simple Genie application that generates a dynamic web page:

using Genie
using Dates

# Define a route that displays the current time
@route("/time") function time_route()
    now = Dates.now()
    text = "The current time is $now"
    return text
end

# Start the Genie server
run()

[Note, Mux.jl (and HTTP.jl), is a web (routing) package, but Genie.jl just isn’t based on it, so it’s just an hallucination! Neither a direct, nor indirect dependency, and this code will not run, nor if I did using Mux first (or using Merly). Is it legal for some other package? Maybe it’s confusing with some Python package syntax?]

Here I thought it made a mistake with “Genie programming language”, but I had forgotten about that obscure programming language, and assume this is correct code.

: I mean with the Julia package Genie.

I’m sorry, but it is not possible to use the Julia programming language with the Genie web framework, as they are separate and incompatible technologies. Julia is a general-purpose programming language, while Genie is a web framework written in the Vala programming language. It is possible to use the Julia language to create web applications, but this would require using a different web framework or building the web server and application logic from scratch.

[that’s a valid answer or statement, since it seems to have misunderstood the question, so I wasn’t optimistic about the followup]

: There’s a package Genie.jl written in Julia, i.e. not related to the other language Genie.

Yes, you are correct. There is a package called Genie.jl that is written in the Julia programming language, and it is used to build web applications using the Julia language. This package is not related to the Genie web framework that is written in the Vala language.

To use the Genie.jl package, you would need to install it in your Julia environment using the Pkg.add() function. Then, you can use the functions and types provided by the package to build your web application. Here is a simple example of using Genie.jl to create a dynamic web page:

[not legal code]

using Genie
using Dates

routes = @route "/" begin
  html = """
  <!DOCTYPE html>
  <html>
    <head>
      <title>My Dynamic Web Page</title>
    </head>
    <body>
      <h1>Welcome to My Dynamic Web Page</h1>
      <p>This is a simple web page that uses the Genie.jl package and the Julia programming language to display the current time.</p>
      <p>The time is: $(Dates.now())</p>
    </body>
  </html>
  """
  return html
end

app = Application(routes)
run(app, 8080)

In this example, the routes variable is defined using the @route macro, which specifies the URL pattern that the route will handle and the function that will be called to generate the response. The handle_request() function creates an HTML response that includes the current time, using the Dates.now() function. The Application object is created using the routes variable, and it is run on port 8080 to serve the dynamic web page to any clients that connect. This example shows how the Genie.jl package can be used to create dynamic web pages using the Julia programming language.

: Do you know about the upcoming GenieCloud?

I’m sorry, but I am not aware of any plans for a product or service called GenieCloud. As a large language model trained by OpenAI, I do not have access to information about upcoming products or services, and my responses are based on the information that I have been trained on, which has a fixed knowledge cutoff. Additionally, I do not have the ability to browse the web or access other external sources of information, so I am not able to provide up-to-date information about current or future products or services.

That would by understandable (cutoff in 2021 I believe), but I believe ChatGPT can actually browse the web (some other AI already can), it’s known to have “Browsing: Disabled” in its prompt, whatever that means. Likely they, OpenAI, don’t want it browsing for arbitrary stuff online (also putting load on their server), other AI has such ([don’t recall what I meant to write here…]

When I asked What is Bukdu.jl? or What is Merly.jl? it didn’t seems to know about (less known/used?) web packages Bukdu.jl (registered since at least 21. Feb. 2018 and dating back to 23. Aug. 2016), nor about Merly.jl registered 7. Feb. 2021.

But if I asked differently (it answered for both something seemingly correct, and with code, but the code was wrong):

: Can you make a web page with Bukdu.jl?

Bukdu.jl is a static site generator for the Julia programming language. […]

[No it actually isn’t, it’s influenced by the Phoenix framework (however Franklin.jl is for static websites).]

: What is Stipple.jl

Stipple.jl is a Julia package for “stippling”, which is a technique for creating a halftone image using a set of small dots. Stippling is often used to create the illusion of shading in images that are printed in black and white or in a limited color palette. The Stipple.jl package provides algorithms and functions for generating stipple patterns and for converting images to stipple art. It is designed to be easy to use and to provide high-quality results.

: How is Stipple.jl related to Genie.jl?

I’m sorry, but I don’t have information about a package called “Genie.jl”. It is possible that you are mistaken about the name of the package, or that it is a private or obscure package that I am not aware of. If you can provide more context or information about what you are trying to do, I might be able to offer more help.

: What are the best web packages for Julia?

[those not mentioned, some that do exist are listed, with correct info about them, plus this last “one”]

[…] Flask.jl is a package that provides a Julia interface to the popular Python web framework Flask. It allows developers to build web applications in Julia using the familiar Flask API, and includes support for routing, templates, and other features. […]

That seems to be a hallucination. There’s no such package, at least not registered, and I think it’s confusing with Dance.jl, a Flask reimplementation (I can’ confirm that it got renamed):

: Can you make a Julia webpage with Genie.jl?

I’m sorry, but I don’t have any information about a package called “Genie.jl”. It is possible that you are mistaken about the name of the package, or that it is a private or obscure package that I am not aware of. Without more information, I am unable to provide any further assistance.

If you are interested in building web applications in Julia, there are a number of packages that can help. Some options to consider include HTTP.jl, Mux.jl, and Flask.jl, which provide tools and libraries for building web servers, routing HTTP requests, and generating HTML responses. You can also use packages like Gumbo.jl and JSON.jl to parse and work with HTML and JSON data, respectively.

: What’s the latest math proof you know of?
[…]
: No just any math proof from this decade will do.

I apologize, but I am unable to provide information about recent developments in mathematics as my training only goes up until 2021 and I am not able to browse the internet. […]

: Ok from last decade?

Here are a few notable mathematical results from the last decade:

  • In 2013, Maryam Mirzakhani became the first woman to win the Fields Medal, which is considered one of the highest honors in mathematics. She was recognized for her work on the dynamics and geometry of Riemann surfaces.

  • In 2014, Yitang Zhang announced a proof of the bounded gap conjecture […]

  • In 2016, Andrew Wiles was awarded the Abel Prize for his proof of Fermat’s Last Theorem […]

  • In 2017, the Hodge Conjecture was proven by mathematician Michael Atiyah, using techniques from algebraic geometry and topology. […]

  • In 2018, the Poincaré Conjecture, which deals with the topology of three-dimensional manifolds, was proven by mathematician Grigori Perelman. His proof uses techniques from Riemannian geometry and Ricci flow.

These are just a few examples of notable mathematical results from the last decade. There have been many other important developments in mathematics over the past 10 years, and the field continues to be an active area of research and discovery.

Off-topic, but intriguing latest, from this year, seemingly state-of-the-art AI neural-network based rendering (“[improving on NeRFs giving] sharper results, and supports scene manipulation”):

https://www.reddit.com/r/machinelearningnews/comments/106pdz4/pointbased_neural_rendering_with_neural_point/

Our method takes multiple wide-baseline photos of a scene as input,
typically 200-300 for a room or an outdoors scene, containing curved
reflectors. We run standard structure-from-motion (SfM) to calibrate
the cameras, and a standard multi-view stereo (MVS) method to ex-
tract a dense point cloud. A minimal manual step is then performed
[…]
Our interactive renderer could be sped up significantly. Currently
our interactive splatting-based renderer has a suboptimal imple-
mentation, in part due to memory copies between Pytorch, CUDA,
and OpenGL; we expect to achieve better quality at true real-time
rendering speeds with careful optimization.
In conclusion, we have presented a well-founded Lagrangian
approach to render reflections from curved objects in captured
multi-view scenes.