Springtime in Paris

There was a time, many years ago, when Paris was a representation in Pemulis's mind of something kind of like freedom or an ideal that was very much aligned with the traditional touristic view of Paris which is at the root of things like Pari shōkōgun (written in the Japanese transliteration because it has allegedly historically mostly affected Japanese tourists manifesting as a sense of strong disappointment when visiting the city, where the feeling was that things weren't quite what they expected and these feelings of disappointment are so strong that, supposedly, those struck by the syndrome experience such symptoms as hallucinations and disorientation and even losing consciousness) and the sadly-entertaining-to-me depictions found in Emily in Paris. Despite being basically as poor as dirt at the time, counting the days till my next pay while jumping the Metro turnstiles, eating only pasta and free mergeuz-frites-salade from my buddies at the kebab shop (probably living no better than George Orwell did during his famous sojourn in the city), this feeling was likely reinforced as the first place that Pemulis lived away from home and then again in subsequent visits staying in nice parts of town, at the right times of the year, with good people, doing fun things, and cetera.

But apparently Paris, surprisingly, is not infallible. Now, good and bad, I may have defined these terms quite clear, no doubt, somehow. Our most recent visit was not at all bad. We had many really nice moments as a family and also at other times when we weren't together. Our trip to the tropical aquarium was really nice as was the dinner us four shared on our last night at a small French restaurant near Montmartre. The Sunday was sunny, about 20 degrees, the cherry blossoms were in bloom, and we ate a nice, quiet lunch in the 3e arrondissement after a very touristy but quite enjoyable boat ride hitting the top sites along the Seine. Joelle ran a good chunk (more than 30 km) of the Paris Marathon route one morning largely before the rest of us had even gotten out of bed, and Helga, despite not getting the colour of Eiffel Tower keychain that she really wanted, I think in the end, while she didn't show it so much, enjoyed our walk up the famous tower and the views that it provided. I had some nice, quiet (emphasis!) visits with up and coming French watchmaking companies (there are many in Paris) and had some interesting conversations with the people behind them.

Something that wasn't quite as nice was the half day (or perhaps more) that I spent finding a paediatrician, taking Heinrich (and paying!) to the doctor, then taking him further north and east to the laboratory that had the right equipment to perform the test he needed, into the absolute worst part of Paris where we witnessed (literally) growing men fighting in the street and one unfortunate soul sitting on the steps of the entrance to the metro smoking a crack pipe. Bienvenue à Paris! The loud dance music that played well into the early morning hours on several nights leading to sleepless nights was not a highlight of the trip for me. Nor was the kids' difficult ummm adaptation to learning how to behave properly (or at least not terribly) while going to bed, walking anywhere, going for meals, etc.

So lies that life is black and white faded far from the foreground on this trip. It was not perfect but it was not terrible either. As I said, we had some really nice moments; but we also had some trying ones. I hope everyone can take with them those good ones and that the bad ones will also fade. But next time I go to Paris, I'll just take Joelle :D

Tour Eiffel!

Aquarium Tropical

Horlogerie française

Touristy Boatride

Sunny Lunch in the 3e

Tasty dinner on the last night

Breaking news

It might come as a surprise to some of you readers out there but not only is there real live AI out there now (of course not philosophically AI like it's an actual being with thoughts or anything like that but the pragmatic/practical definition of AI with emphasis on the artificial where a system can -- and I guarantee you it can -- pass the Turing Test) but also I recently turned 42.

Unless you've been living under a rock for the past few months, you know about ChatGPT and how it's kind of a big deal. Of course it's only a "large language model" and all it really does in the end is (essentially) predict probable words (tokens) given what has been generated before and the context. But this is a moment... It's eery how much it can do and really shows artificial intelligence and can actually be really helpful.

While it can do silly things like give answers that make sense to like "write a song about blogging in the style of Taylor Swift" or "give a proof that there are an infinite number of prime numbers making sure that all sentences rhyme", it is also extremely useful in day to day work. It's really like having an intern working for you. For my work where you might spend a day figuring out how to get esoteric thing working -- like getting one technology to work with another -- you just ask ChatGPT and it really almost always solves it. And it's literally like having an intern in that it will fix its mistakes. You ask for something, it comes back with a pretty good first draft, you see that there's one issue or a case that was overlooked, you point that out, and it comes back with an improved version. It's crazy.

And it seems to somehow understand humour (to a certain extent) as well. It can look at images, understand what's going on, and then explain them. Look at this real example... Here is a picture I shared with a friend:

It is a corny Physics joke. My friend showed the image to GPT, and said "please explain this joke". Here is the output:

I mean... that's scary, right?

The important thing to remember is that all of this is "learned" simply by analyzing billions and billions of images and texts from the internet and finding (in a very structured and complicated way) parameters that help to predict the next word fragment (by using word fragments as "tokens" instead of full words, the model can even invent new words that it has never seen before). So it's finding statistical correlations that exist throughout all the writing, etc. that has been produced by millions of different people over a huge amount of time in different domains and it's been somehow all sucked up into a model.

Now the question is, why does this work? One of the ways that the model works is that it doesn't always predict the most probable next token. That would not tend to result in very interesting output. Instead, it samples from the probability distribution over tokens that is setup given the context so far. And though the interface to ChatGPT abstracts all this away, one can tell the model how "random" it should be by changing the "temperature" of the probabilities so that we can explode or diminish the large probabilities. With a low temperature, we explode the probabilities and the model will be very boring and be extremely likely to predict the most common token next. With a high temperature, it's the opposite and all the probabilities get closer to each other and it becomes much more random.

As for why it's able to respond in this conversational way, that is a second step in training the model where the parameters are adjusted based on training with "humans in the loop". At first the model is likely very unconstrained and then you slowly teach it to behave in a certain way. So you ask it to do something, and if it's close to what you want, you give it positive feedback, and if it's not at all how you want it to respond, then you give it negative feedback. Obviously the way you approach this is very specific and technical and I don't know the exact details, but eventually the model settles into a state where it's able to answer as you see above. There are other things that I'm sure they are doing that involve pre-processing the text. So if you asked something that they don't want to talk about like crime or whatever, then they might have another model to predict that it's something we don't want the main model to even see so they filter it out and give a canned response back like "I don't know too much about that".

So anyways, what does it actually mean? I think like many things, in the end, it's a tool. Noam Chomsky has a good opinion piece in the NYT recently: "The False Promise of ChatGPT". Basically, it's not intelligence; but, we already knew that, right? I should note here that (obviously, based on what's written above and also based on the wide range of learning material) large language models like ChatGPT make mistakes and often they make mistakes that no human that had actual intelligence would make. Things like inventing stuff that never happened or being confused about timelines or just really inventing concepts or places. Of course this can happen because it's just drawing tokens from a probability distribution! 

But I was talking with a friend recently about how since large language models just look at what's happened before, they completely rely on work that humans have done and just essentially regurgitate it in a different way, so they could never come up with something new. But he responded with: isn't that exactly what humans do? And maybe he's right. Like the series "Everything is a Remix" which explains that all new art/creation is necessarily based on things that happened before. So who knows... maybe we're all doomed?