In the beginning (A Story of Lorito and Parrot)

In the beginning...

...there was Parrot.  And it was magical.  But it was slow and would not stand still, so it was hard to target.  Then there were calls amongst those near Parrot for speed and efficiency.  Thus, Lorito was imagined.  It was to be leaner, faster and more stable, but still be able to handle almost all the magic of Parrot.  Ideas were thrown around, and much hand waving ensued.  Then, an idea of limiting the core to 20 opcodes was thrown out.  This piqued my interest, so new code was written, stubbs were created, and a project on github created.  Code was written fast and furious and a vision appeared.  I began to dig at the surface of the issues that Parrot exhibited today.  Implementation details began to form around avoiding the mistakes of the past, as they were described to me.  Soon, a mantra was formed: Less Magic == More Magic.  Simplicity and efficiency can be optimized later, but complexity today remains tomorrow.

First, I made the opcodes single sized.  8 bytes, and only 8 bytes ever.  1 for the opcode, 3 for the 3 arguments (destination, source 1 and 2), and 4 bytes for the immediate value.  Quickly, I meet resistance.  "We need far more than 255 registers" was the cry I heard.  I took note, contemplated doubling opcode size, which could afford 4k of registers.  Then I heard the reason why they needed more registers.  "Because that's what's produced today."  When I dug the surface of the claim, it appeared that there is no register allocation scheme.  That many were used because they could be.  I said to myself "I have a hard time believing that at one point in so many programs, that a function call would need to track more than 255 registers at one time."  So the decision stayed.

Then I said that all PMCs are pointers and have no inherent structure.  There would be no way to ask, politely or otherwise, the PMC itself what it looked like.  The PMC could have a gate keeper, but forcing that seems like the wrong solution to the problem.  Why, I was asked.  That will make them flexible, I respond.  I have removed no functionality, no features lost.  Everything that has been done before can still be done.  Do we complain that machines have blocks of memory that we access by address and not by name?  But now, you can manipulate, store and load any data of any size and requirements into a PMC as though it is merely memory, because it is.  Since the PMC is always the same size, and it is the only thing that points to it's memory, being able to rearrange after garbage has been collected for the time is trivial.

Then, upon the advice of elders, I stopped treating PMCs with a level of reverence that objects do not enjoy. They are the same, I declared.  No difference.  Objects are PMCs, and PMCs are objects.  No more tables of v, they are all methods to be lookuped and called.  Perhaps, I whispered to myself, we still need some special functions.  Simple, I replied, for the information you can ask of all PMCs, make those special functions special.  But do not let others interfere, and make their numbers small.

I surveyed my objects around, and saw my scatterings.  Registers put into boxes, segments of consts, datas and code.  In inspiration, I promoted them all to the same level.  Special PMCs, yes, but PMCs none the less.  To be interacted with in an identical matter as all other objects.

Then I noticed the registers that spoke words.  What use are they as special?  But, how do we efficiently do lookups on objects without them?  Then someone spoke of symbols.  What are these symbols, I asked.  He showed me a world where comparison was cheap of large volumes text.  Symbols become the new register and we gain performance for lookuped objects and regular manipulation of these strings become objects, like everything else.

I create a container that contained everything Lorito needed to know about what it was doing.  It had the registers, the code it's working on, the arguments it has and returns, on the constants it has access too.  It has no requirement to return where it came from, so that will relieve some concerns about being able to do magic called CPS.  This context object then leaves the details of how the operations actually take place inside an interpreter for Lorito.  Later, after a braindumping, the idea was reinforced by those in attendance.  Thankfully, brains intact.

How can you call methods of objects, some asked.  Here's an idea, called P&W, another one told me.  Interesting, it was, but perhaps was still too complex since it dictated inheritance.  What if there is no need for inheritance?  What if inheritance is complex?  Let the object deal with it all.  Make a special operation called lookup.  When it's called, call the object's special lookup method and pass it whatever object it handed us in the beginning.  It shall let everyone talk to everyone, and allow all any possible complexities to be possible.

Lorito has limited operations, and the instructions are limited in size.  The way to call methods will have to change.  No longer can we pass all of the arguments in one operation.  So a context is created to contain these arguments and other details, and the arguments are added to this in separate operations, not unlike a stack of trays.  Then, when the method is done, it can return any information in the same list.  But this method is not without controversy.  Efficiency, speed, compatibility are the concerns I hear.  I cannot speak to any of the three, but I believed it was my only choice with my decisions.  But the calls to those not inside the sphere of Lorito, they can be simplified.  These outside functions can be treated inside Lorito exactly like Lorito methods.  Then, once this function is called, it is passed the context I already have and it can do it's magic.  A nice, simple way of interacting with those that are not Lorito.  Sadly, the concerns of others cannot be satisfied for some time, but I persevere.

Parrot has a difficult time talking to itself.  It can ask itself a question and wait for an answer, what it likes to call it's innerloop, but it's slow and prone to problems.  Others say that Lorito will eliminate this problem while I am less optimistic.  Lorito may alleviate the problem, but there is a lot of magic needed to eliminate it.  So, I save the solution for another day.

I wondered, can Lorito exist without goto or call?  I stood and thought, and decided that no, it can't.  Not unless we make a huge, flat map of all the instructions that Lorito knows about at any one time.  So I kept both types of operations and made the available code available to Lorito self contained.  I like things that are self contained, it's easier to handle and simpler to work with.  So I made the sheets of instructions individual for each method.

But I worried, I needed to justify my operations to anyone that uses Lorito.  So I wrote instructions on how to use each of my 44 operations and justified the existence of each.

As I looked over what I had accomplished, myself asked me a question.  What if all this is not what others what?  That's okay, I assure myself.  It's not the destination, it's the journey.  I understand the decisions I made are a departure from what Parrot is today, but this is about how I would create Lorito to support Parrot.  I believe earnestly that anything Parrot can do, Lorito can do and then some thanks to the flexibility and simplicity of the decisions made.  Its future is uncertain, but I will continue to work on it until there is no need for it, either by others or myself.

Less Magic == More Magic

(Yea, probably a bad idea.  But fun, I must admit)

My Little Lorito World

It's been a long time coming, six months in fact, that I intended to start blogging again.  And by again, I mean actually try and do a good job at it.  We'll see.

But, with some prodding by cotto, I've decided, hey, why not, let's review atrodo's little Lorito world.  One could argue that I should step back and explain what I'm talking about it, and why I'm interested.  But I won't, since it'll take too long and I don't expect people that don't know what I'm talking about to be here yet.  If you are, sorry.

But, real quick.  Parrot is a virtual machine, not unlike the JVM or .Net.  It is aimed at more dynamic languages, but I actually have an interest in static languages on Parrot.  Lorito is a hand waving term for a group of refactoring the Parrot project is aiming for.  I started a Lorito prototype back in July-ish after some very basic guidelines were posted to parrot-dev by allison.  Since then, I've been working on it (along with Hordes of Tarshish and isparrotfastyet.com, not to mention $dayjob) in an attempt to reconcile the visions of other parrot developers and my own.  I will be saving my current Lorito vision for another post (which is apparently a narrative, huh).  But at issue today is the meeting from last Thursday between some of the core parrot.  I would have tried to attend, but Portland is no where close to where I live.

There were 10 points in cotto's notes, so I'll hit my highlights.  The original notes can be found on a gist I created, and went on to edit with my notes (this is the current version).

Point 1: There is only context.

This is kind of a muddy idea, leaving a few details that I suspect are important, out.  But it's one I can get on board with.  I will, for the time being, keep an interpreter around, but change it's function and the concept behind it.  From now on, the interpreter will be in charge of executing bytecode as well as holding the pointers to the loaded files, symbols and the PMC heaps (which, not implemented in Lorito yet).  The context, for all intents and purposes, holds everything else.

The rest, however, worries me.  It implies several key things that I'm not sure I'm on board with.  First, that the MOP is so tightly integrated with Lorito, that Lorito cannot be used without using the Lorito MOP.  Now, maybe that's desirable to some extent, but the idea of Parrot is to allow a large cross section of languages to target it.  If I have to use Lorito's MOP, no matter how flexible it sounds, that sounds contrary to these ideals.

The other worry some part is that setting a program counter makes it jump to C.  To me, jumping between the boundaries should be well contained and explicit.  The C code is it's own context, it does not share it with the VM.  There is also, I think, a lot of good security intentions that will make security more difficult down the line.  But that's my gut telling me that.

Point 2: Register Count

That's cool.  Although, I had already set my limit to 255, since that's what can fit in a byte to keep the opcode length down.  But what really concerns me, is the integration of the MOP with the registers.  I just don't understand.


Point 3: Alloc as an op


I've already done this.  It's my new op.  To me, PMCs are blobs of memory, so that way you don't have two ways of doing essential the same thing.  Some may argue that a PMC needs structure, but I don't see why that's not true with my method.  Then again, I always take the encapsulation method.  Outside code should not be able to peel back the veil and inspect my data structure.  This actually leads me to considering a seal function on PMCs, so only a particular code segment can do store or load operations on a PMC.  But that has some developing to do.

Point 4: ffi

This is important, I must agree.  But I also think this might be outside the bounds of Lorito.  Personally, and I've gotten some feedback about this position, I think Lorito bytecode should have one way to interact with anything from the outside world.  This means Lorito does not interact with nci or ffi directly.  Instead, it delegates that complexity to other code.  Yes, I know this sounds inefficient, but I have heard whiteknight and chromatic repeat over and over that PCC (Parrot Calling Convention) is slow.  Really slow.  So I abandoned anything that resembled PCC for two reasons.  First, for the above being slow.  Second, because I don't have the opcode length to.  Instead, all Lorito calls are made by pushing arguments (all of which are PMCs) onto a stack in the context.  If it's a Lorito method, it can pop the arguments off.  If it's a C function, the C function can pop the arguments off.  In the case of nci or ffi, it can be thunk that massages the arguments into C types, and call the actual C function, arguments in tact.  The big advantage of this is the simplicity.  If you're interacting with Lorito, you can get the arguments easily with a (inlinable) function call.  If you're calling nci or ffi, you still have to do your thing.  But you would have to do it anyways.  So what I've done is made the simple case simple (and in theory, faster) and the complex case possible.  The PCC way is overly complex, and is always complex, even for the simple case.

Point 5: int sizes

Haven't thought a lot about this.  Honestly, I've been ignoring the issue.

Point 6: Don't fear the REPR

Seems completely possible to me.  No matter if the REPR is at the Lorito level, or at a higher level, this seems worthwhile.

Point 7: CPS and Context

Exactly what I do today.  Hurray!

Point 8: Awaiting clarification.

Point 9: Pass

(I know it looks like I'm giving up, but really, there's nothing for me to talk about)

Point A: Strings, not Symbols, are PMCs

That sums it up.  The S register in my Lorito are symbols.  Not Strings.  Strings are high level constructs and deserve to be elevated to PMC status.  But low level symbols are a highly useful and time saving construct.  I've added no functionality to manipulate symbols.  They are there to load into registers and compare.  That is about all they can do.

Conclusion:

I was hoping for concrete answers from last Thursday, and I got some, but I also got a lot more possible directions.  So, I am going to keep on doing what I've been doing, and try and reconcile ideas as the come along and advocating what I've done so far.  Not sure where this is going to lead me and Parrot, but the possibilities are interesting to watch unfold.

And wow, that's a lot of text for such a short topic.