Planet Perl Six is an aggregator of select Perl 6 related blogs. The list of contributors changes periodically.
Planet Perl Six provides its aggregated feeds in Atom, RSS 2.0 and RSS 1.0, and its blogroll in FOAF and OPML
69 years ago today off the coast of western Australia, two ships engaged each other in battle — the German Kormoran one and the Australian HMAS Sydney. The final outcome of the battle was that both ships were destroyed.
There's a lot of nifty details about the signaling before the battle, which I will not relay here. But here's one thing I didn't know before: you have to flag that you're going into battle according to international laws of war (and there is speculation whether the Germans really did this). To me as an outsider, it seems like a surprisingly honest thing to have to do in a battle situation.
â¦
Seeking to patch up .trans to handle regexes today, I failed in unexpected and interesting ways.
Having distracted myself during $dayjob with thoughts of how to go about implementing regexes in .trans, I finally dug into the task in the evening. This is what I had concluded while just thinking about it:
Str as hash keys right now in Rakudo. No worries, an array works in this case.This is what I learned by implementing it:
.trans algorithm operates: it keeps all the "next match" positions up to date for all things it's matching on.) Ended up with an array of pairs, the keys of which were also pairs. Hello, cadr.The last guttural sound there was me giving up due to an exceptional amount of minutia that I hadn't anticipated in the design phase. ☺
TimToady++ on the channel understood immediately what the trouble might be.
<masak> today, I've tried to patch .trans for handling regexes.
giving up for the day -- it was more complicated than I
imagined it would.
<masak> time to blog about the unexpected overwhelm fail. :)
<TimToady> masak: that will be very difficult unless you know how to
hook into the LTM implied by .trans
<masak> TimToady: yeah.
<TimToady> espcially since rakudo doesn't really do LTM yet
<masak> TimToady: the current .trans impl does it right for constant
strings.
<jnthn> The current Rakudo does LTM right in some cases for constant
strings. :)
<masak> TimToady: but... I found I had to special-case regexes, and
then there were a lot of "interesting" corner cases across
the boundary.
<masak> I think I'd be better off attacking the problem after hiding
some of that complexity first.
<masak> essentially runtime-polymorphing on constant strings and regexes.
And that, in a nutshell, is what I intend to do if-and-when I attack this problem again. Need to build a little LTM engine that hides the complexity of handling both constant strings and regexes in the key position of the pairs sent in to .trans.
That should take care of another problem I had with the code: it wasn't fun to read. Putting an LTM engine as an API between the .trans code and the complexity should help.
54 years ago today — and yes, we do seem to have a bit of a Communism theme going — then Soviet premier Nikita Khrushchev said to a group of Western ambassadors "We will bury you!"
Be that as it may, Google Translate doesn't suggest "dig you in", it says "bury".
Khrushchev appears to have been a hot-tempered Soviet leader. The kind of guy who sticks his chin out, bangs his shoe on the table and says "Kuzka's mother" to president Nixon. A few years later, Khrushchev mentions the "bury you" comment.
The contrast between "the victory of the proletariat" and the two silent revolutions from yesterday and day before yesterday, is nothing short of stark.
I also found this lovely little Time article about the "bury you" incident. It describes Krushchev working himself up into a rage and tramples around on several diplomatic feet.
His voice was scornful as he added: "But we, Mister Capitalists, we are beginning to understand your methods."
By this time, the diplomats—who, in turn, have come to understand Mister Khrushchev's methods—had already left the room.
â¦
Didn't get started on a .trans patch today. $WORK got in the way, and before I knew it, it was time to cut a Rakudo release.
Right now I'm running the spectests one last time, taking the chance to write this blog post. It hasn't been very taxing to make the release this time either — especially not compared to last year, when the clock was ticking — but I figure I should pace myself and save the .trans patch, or whatever, for tomorrow. Releasing Rakudo is definitely worth one day of November blogging.
I'll just add in passing that, a little over half-way through this Month of November, I'm having no trouble finding things to do every day. I expect to reach the end of the month feeling "what? no more November?" rather than "phew! finally over!".
Well, I'll probably feel a little bit of both. 哈哈 But my point is that there seems to be a lot to fill this month with. My main concern is that I might get "stuck" in any of my projects and devote too much time to that, to the detriment of the others.
And that's rather nice, as problems go. I'm glad I decided to do another month of November blogging.
21 years ago today, riot police suppressed a peaceful student demonstration in Prague, capital of Czechoslovakia.
This is all very reminiscent of the Tiananmen Square protests (å…四事件) earlier that year, which also started with a grief-based demonstration, that time over the much more recent death of Hu Yaobang, a well-liked reform-friendly politician.
Anyway, back in November and Czechoslovakia, the peaceful demonstration was brutally suppressed. This image has become a classic symbol of the stark contrast — flowers against shields and guns.
The following detail reveals that the secret police may have had a role in orchestrating the events that followed:
Things snowballed from there.
I'm trying to think of something insightful here, about how the peaceful overturning of Communist states by the people is perhaps the ultimate sign that those states (and, by extention, the Marxism-Leninism practiced in those states) weren't working well. Then again, one might argue that details such as rampant nepotism, massive deportations, or barbed wire to prevent people from escaping the country, were just as clear signs.
â¦
Last night I left off with an error in Text::Markup::Wiki::MediaWiki that seemed to indicate that something nasty was sent into Str.trans.
Well, there wasn't anything wrong with what was sent in. The call looks like this:
my $cleaned_of_whitespace = $trimmed.trans( [ /\s+/ => ' ' ] );
Rakudo has no idea what to do with the regex /\s+/ among the arguments. It used to obviously, since November once ran this line of code fine. But it doesn't any more. Yet another ng regression. I reported this as rakudobug #79366.
I'd like to fix this, preferably before the monthly release of Rakudo tomorrow. We'll see if I have time to write up a patch. (Unless someone beats me to it; hint, hint.) I don't think it'd be that difficult to fit into the existing algorithm.
tl;dr;; version: arbitrary API limitations do more harm than good, even if meant well in the first place.
Most advanced libraries that help you with date calculations have separate data types for a point in time, and a time span. That's because those two concepts actually have different semantics. It doesn't make sense to add two points in time, but it does make sense to add two durations, or add a duration to a point in time.
In Perl 6, those two types are called
Instant and
Duration. And obviously it makes sense to multiply a Duration
with a number, but it doesn't make sense to multiply two Durations, or take a
power of a Duration. Right?
That's the opinion that both the Perl 6 specification and the implementation in Rakudo reflected. Until recently, when somebody started to actually use it.
That's when the real world struck back. Carl Mäsak did some timings, and then calculated averages and standard deviations. And for calculating standard deviations, you actually have to square those durations, add them up, and then calculate the square root.
So this perfectly legitimate use case shows that multiplication (and also
exponentiation) are perfectly fine operations on Durations. Likewise the
current specification disallows division of two Durations. Why? It's perfectly
fine to ask for the ratio of two time spans. How much longer (or shorter) are
my meetings with my current boss, compared to those with my previous boss?
That's the question that Duration / Duration answers.
So, the real world taught me that putting restrictions on the allowed operations is a bad idea. It was meant well, it was supposed to catch operations that don't made sense to the designer, and presumably would catch some error that a confused beginner might make. But in the end it did more harm than good.
Currently the Duration class stores a number, and
re-dispatches all operations to that number, forbidding some of them. Having
learned my lesson, I suggest we get rid of it, and have Instant -
Instant return that number straight away. If some day we want to
add functionality to Duration, we can still create that class as
a subclass of the number.
22 years ago today, Estonia took its first step towards independence, after having been continually occupied since 1940 by first the Soviet Union, then by Nazi Germany, and then once again by the Soviet Union.
In the climate of Gorbachev's Perestroika, however, Estonia could begin taking steps towards regaining independence.
Sovereignty isn't the same as independence. This was only the first step in a chain of events that didn't end until 1994 with Russian (no longer Soviet) armed forces withdrawing from Estonia. It's the steps in the middle that are awesome, however. A new parliament was formed through a grassroots movement:
Estonia was unusual among the Baltic states both in the leading role it took in negotiating its independence, and the bloodlessness by which it happened.
â¦
Ok, remember how yesterday, I left off with a "Cannot modify readonly value" error? Turns out the reason for the error is at least of mild interest. Here's a simplified version of what happened in November (the wiki engine):
$ perl6 -e 'sub foo($x is rw = "OH HAI") { $x .= lc; say $x };
> my $y = "TESTING"; foo($y);
> foo()'
testing
Cannot modify readonly value
in '&infix:<=>' at line 1
in 'foo' at line 1
in main program body at line 1
See what's happening there? Our parameter $x is declared to be modifiable (is rw), but we can only actually modify it if we send in a variable from the outside. If we let the parameter assume its provided default ("OH HAI"), it no longer allows modification.
Is this correct? I can argue both ways on that point, I guess.
On one hand, assigning the default "OH HAI" to $x could be seen as equivalent to passing in the literal "OH HAI" to &foo. In this case, we do expect any modification to blog up with a "Cannot modify readonly value" error, because what we've bound to $x isn't really a variable, it's a constant literal.
On the other hand, we did specify that $x be rw, and we're not really passing in "OH HAI", we're providing it as a default. So what's stopping Rakudo from assigning (rather than binding) "OH HAI" to $x, allowing for subsequent modification?
I'm currently leaning towards the other hand. ☺ But I'm betting jnthn++ will have useful input to offer, since he's the signature binder guy.
Here's an even smaller example of the same underlying issue:
$ perl6 -e 'sub foo($x? is rw) { $x = "OH HAI" }; foo()'
Cannot modify readonly value
in '&infix:<=>' at line 1
in 'foo' at line 1
in main program body at line 1
That is, when we don't pass in a value, it doesn't matter that $x has been declared is rw; we still can't modify it.
As a temporary workaround, is copy seems to solve all the perceived issues.
$ perl6 -e 'sub foo($x is copy = "OH HAI") { $x .= lc; say $x };
> my $y = "TESTING"; foo($y);
> foo()'
testing
oh hai
So maybe the answer to my worries is simply "use is copy and stop whining!". Actually, now that I look at the original failing code, it's pretty clear to me tha is copy is a better fit in this case. But I'm still curious whether we can do better when is rw interacts with defaults. [Update: This is now rakudobug #79288.] [Update on the update: By the decree of a spec patch, this discussion is now moot.]
Applying that workaround I find, unsurprisingly, more bitrot-related runtime errors that require fixing. Some $file ~~ :e need to be turned into $file.IO ~~ :e. I end today's bushwhacking on a faulty Str.trans call somewhere in Text::Markup::Wiki::MediaWiki.
102 years ago today, Empress Dowager Cixi (慈禧太åŽ) passed away. She had ruled China for 47 years from "behind the curtains", going against imperial tradition forbidding women to engage in politics.
She has a generally bad reputation in the scrolls of history.
Nevertheless, her life is full of plotting and intrigue, something which no doubt tickles the mind of the historically curious. During her nephew Zaitian (載湉)'s rule, Cixi had him put in house arrest for nine years. He died, 37 years old, on the day before Cixi passed away, apparently from arsenic poisoning.
â¦
To my surprise, the thing I meant to do today (converting November to work on Rakudo ng) had already been started. By me, no less. I had forgotten that.
Hm. Seems I did some initial work on it in relation to this post, and then continued that work a few days later.
So where did that put me? I run make in the ng-compat branch of November to find out.
Ah. #73912 bites, and we get this bogus error message:
===SORRY!===
Illegal redeclaration of symbol 'November'
Luckily, at some point pmichaud++ told me how to deal with this. (Can't find it in the backlog, but I remember him telling me.) The trick is to stub the class before you start pulling in any modules that use that namespace:
class November { ... }
That worked! Now November gets as far as pulling in HTML::Template which, amazingly, still compiles. These Makefiles are old enough that I need to manage PERL6LIB by hand — how barbaric.
Oh, and it turned out I needed to update the script wiki, responsible for actually running the November application. It had bitrotted from previous changes in module naming done in that branch.
Running perl6 wiki, this is what I get now:
Cannot modify readonly value
in '&infix:<=>' at line 1
in 'November::view_page' at line 53:lib/November.pm
in <anon> at line 37:lib/November.pm
in 'Dispatcher::Rule::apply' at line 44:lib/Dispatcher/Rule.pm
in 'Dispatcher::dispatch' at line 37:lib/Dispatcher.pm
in 'November::handle_request' at line 49:lib/November.pm
in main program body at line 17:wiki
Probably Rakudo has become a bit more stringent with readonly things since alpha. This will provide a good starting point for tomorrow's investigations.
I'm excited to announce that Parrot Foundation and The Perl Foundation have been accepted as organizations in Google Code-In 2010!
Google Code-In is a contest, similar to Google Summer of Code, where Google pays students aged 13-18 to do tasks designed by open source communities, while learning about open source. Google pays for the work to be done, and we get new members to our communities, while students learn useful skills. It is a big win for everyone.
In 2010, Google Summer of Code was a great success for Perl and Parrot. We got amazing new features in Parrot, Perl 5 and Perl 6 . In 2009, we had similarly spectacular results.
For the students, the benefits are huge. They get mentored by some of the best minds in open source and get "street cred" in the community. This contest also acts as a stepping stone for Google Summer of Code, so students that excel at Code-In will most likely be sought after for future Google Summer of Code involvement. It's also fantastic experience to put on a résumé. I see many Google Summer of Code students get snapped up by respected companies, or accepted to prestigious academic institutions.
The more well-documented tasks we have before that, the more students we will have the potential to attract. I can attest that these kind of contests attract some of the smartest students in the world, so the Perl and Parrot communities have much to gain by being involved.
I expect great results for Code-In as well, but we need your help. The Google Code-In contest opens up for students on:
122 years ago today, reporter Nellie Bly embarked on a journey around the world, attempting to turn the fictional Around the World in Eighty Days (published 15 years prior) into fact.
I know "delayed by several weeks" must seem like an absurd concept for people growing up in the Age of the Tweet.
Anyway, she made it. 72 days. Nellie++.
â¦
As of today, it's possible to call closures in Yapsi. This thanks to the pioneering work done by patrickas++, the few gaps of which I helped fill in some hours into the night last night.
The work turned out to decompose into two steps:
First we made the parser distinguish between non-immediate and immediate blocks. We used to have only the latter kind; now we only consider statement-level blocks immediate.
Then we added the ability to call a closure. Voilà !
This is good news, because this lays the groundwork for subroutines in Yapsi, a big milestone. In fact, I'm going to go attempt subroutines now.
Perl 6 aims to give you sane defaults, but to allow you to hotswap in your own custom infrastructure when needed. Perhaps the most famous example of this is the grammar, which can be extended with slangs (or sub-languages) at compile-time, essentially extending the Perl 6 language into something more fitting to your needs.
Another example is the OO meta-model, that allows you to create new types of OO objects, beyond the normal class/role/grammar triplet in Perl 6. I played a bit with both grammar and meta-model in the post Extending the syntax of Perl 6, arriving at the conclusion that while it's possible, it's not easy today, and you have to rummage a bit in the internals of Rakudo. You shouldn't need in "real" Perl 6, because there would be hooks to do this from the comfort of your module.
Today, I tackled another part that's supposed to be overridable eventually: the multi-dispatcher. Just a quick reminder to everyone about what multi dispatch is:
multi sub foo(Str $x) { say "You passed in a Str" }
multi sub foo(Num $x) { say "You passed in a Num" }
multi sub foo() { say "You didn't pass in anything" }
Or, in words, multi dispatch allows you to have several routines with the same short name, but differing in their long name, which includes the routine signature. When you make the call to a routine of which there exist several multi variants, a part of Perl 6 starts sorting them out, seeing if the arguments you passed in matches the parameters in the multi signatures.
That part is called the dispatcher. The dispatcher reaches one of the following three conclusions:
multis match. An exception is thrown.multis matches (most narrowly). Happiness! We go ahead and complete the call to that particular variant.multis match. An exception is thrown.Thus, all the interesting stuff takes place in the habitable oasis between the desert of no matching candidates and too many matching candidates. The irrigation provided by multi dispatch is certainly bountiful enough to allow for some rich semantics.
Here are some more examples:
# Dispatching on number of arguments
multi sub a($x) { say "1" }
multi sub a($x, $y) { say "2" }
a(42); # "1"
a("4", "2"); # "2"
# Dispatch on typing versus no typing
multi sub b(Str $x) { say "Str" }
multi sub b( $x) { say "something else" }
b("OH HAI"); # "Str"
b(42); # "something else"
# Dispatching on type narrowness
multi sub c(Numeric $x) { say "Numeric" }
multi sub c(Rat $x) { say "Rat" }
multi sub c(Int $x) { say "Int" }
c(4 + 2i); # "Numeric"
c(4/3); # "Rat"
c(4); # "Int"
In this last case, even though all three calls pass a Numeric argument, the latter two get dispatched to the more specific variants simply because they are more specific. The first call gets dispatched to the general Numeric variant because none of the specific ones match. In each case, the dispatcher is left with exactly one candidate that matches most narrowly.
However, we can create trouble for ourselves by making a set of multi variants neither of which wins out in some case:
multi sub trouble($x, Str $y) { say "something and Str" }
multi sub trouble(Str $x, $y) { say "Str and something" }
trouble(42, "mmm, pie"); # "something and Str"
trouble("OH", "HAI"); # BAM! ambiguous dispatch!
This is the "more than one of the multis match" case. The Perl 6 dispatcher discovers that even after attempts to filter out less narrow candidates, two still remain in the end. So it gives up.
Today, I decided to override that behavior to choose randomly between candidates that tie. I wanted the following (more obviously ambiguous) dispatch to work:
multi sub cointoss() { say "Heads" }
multi sub cointoss() { say "Tails" }
cointoss() for ^10;
The above should produce ten random selections of "Heads" or "Tails". The trouble example should also work, and randomly select which method it calls.
Now this might not be a very useful semantics, but it's at least very easy to understand. And, as it turns out, it was easy to hack into Rakudo.
* jnthn temporarily confiscates masak++'s Rakudo commit bit
<masak> no worries. :)
<masak> I'm not in an iconoclastic mood.
<masak> I actually *like* the way the multi dispatch works.
Needless to say, overriding the default dispatcher in Rakudo is not just a matter of accessing some variable in userspace (yet). Still, the change I had to make to Rakudo was delightfully small:
commit 757669370b27ef1c43517bdf1af8d964d6cb60d7
Author: Carl Masak <[email protected]>
Date: Sun Nov 14 17:21:09 2010 +0100
[perl6multisub.pmc] randomly resolve tied multis
diff --git a/src/pmc/perl6multisub.pmc b/src/pmc/perl6multisub.pmc
index 72bcd0b..540c155 100644
--- a/src/pmc/perl6multisub.pmc
+++ b/src/pmc/perl6multisub.pmc
@@ -691,16 +691,9 @@ static PMC* do_dispatch(PARROT_INTERP, PMC *self, candidate_info **candidates, P
signatures);
}
else {
- /* Get signatures of ambiguous candidates. */
- STRING *signatures = Parrot_str_new(interp, "", 0);
- INTVAL i;
- for (i = 0; i < possibles_count; i++)
- signatures = dump_signature(interp, signatures, possibles[i]->sub);
-
+ PMC *result = possibles[rand() % possibles_count]->sub;
mem_sys_free(possibles);
- Parrot_ex_throw_from_c_args(interp, next, 1,
- "Ambiguous dispatch to multi '%Ss'. Ambiguous candidates had signatures:\n%Ss",
- VTABLE_get_string(interp, candidates[0]->sub), signatures);
+ return result;
}
}
else {
Now this works in my local Rakudo:
$ ./perl6 -e 'multi cointoss() { say "Heads" };
> multi cointoss() { say "Tails" };
> cointoss() for ^10'
Tails
Tails
Tails
Heads
Heads
Heads
Heads
Heads
Tails
Tails
There we go. Desired result, and the only thing needed was changing some C code. The change to the C code was so easy, even I could do it. Of course, the big win (just as in the "extending the syntax" post) would be to be able to make such a change from within user land. And we're not there yet.
When I mentioned that to jnthn, he got a slightly worried look. He pointed out that almost everything is based on multi dispatch in Perl 6, and if we can't make any assumptions about the dispatch mechanism, we might not be able to optimize the Perl 6 as much as we'd like to.
I think that's a valid objection, but not a show-stopper. People who want to switch in their own dispatcher probably won't mind if optimizations are turned off, as long as they can do it. And if the modification is restricted to just the lexical scope or the module where it's done, the effect might not be too disastrous.
Anyway, this was today's little experiment.
35 years ago today, the steamship Yarmouth Castle, on a pleasure cruise from Miami to the Bahamas, caught fire and sank.
The ship was made of wood, and was freshly painted, so the fire spread quickly.
87 people went down with the ship, and three later died while treated at hospitals. The disaster brought on changes in the international Safety of Life at Sea law. One of the changes was "do not build passenger ships out of wood".
â¦
I've been here and there today, nibbling at stuff. Did a little more work on pls, cleared out my queue of pending rakudobugs, and patched an LTA error message in Rakudo.
But the chunk of "actual" November work today was restoring Shrdlu to the living. The commit summarizes what needed doing:
*. (I submitted this as [perl #79166], because I think it's ridiculous that the prior content of a variable would determine whether the assignment "succeeds".)rules really wanted to be tokens. (Grammars are hard. Someone should give a course or something. I've learned a bit since I wrote that code.)>>~~<< needs parens around lhs and rhs. (Generally, hyperops have "transparent" precedence, i.e. the same as the operator they're meta-ing. Since ~~ would require parens, so does >>~~<<.)Shrdlu is a really nice project that I will tell you all about some day. Also, I'll start actually hacking on it some day... after I've grokked what that Terry Winograd was doing.
68 years ago today, the Australian government approved a second attack on the enemy host — because the first one hadn't been particularly successful — in what was to become known as the "(Great) Emu War".
You read that correctly. The enemy that Australia was fighting was the large, silly relative to the ostrich known as the emu. About twenty thousand of them were causing enough trouble for the Australian farmers, that it was decided that military action was needed.
Guns against emus. I'm not making this up. Anyway, the first attack wasn't the amazing success one might expect when the Royal Australian Artillery with frigging machine guns moves against a group of flightless birds.
In the words of ornithologist Dominic Serventy:
Major Meredith also expresses his clear disappointment at the birds' sturdy builds:
I was going to say that in the century of gene modification, someone is bound to build an army of armed emus sooner or later. But I'm momentarily stunned by the comparison of emus to Zulus.
Anyway, the second attack was more successful, resulting in about a thousand direct kills, and about 2,500 birds dying from bullet injuries. Allegedly, 100 of the emu skins were collected, their feathers used to make hats for light horsemen.
â¦
A decent test suite can help you catch simple programming errors. An excellent test suite helps you with the design.
A decent test suite tells you "you missed a spot". An excellent test suite tells you "no, that's not it — I remember more of the design than you do". And then, when you get it mostly right, it tells you "you missed a spot". Or, put differently, you're not done until both you and the tests say it's fine. It's bloody frustrating, but it really helps, too.
Today, I had that kind of battle with pls. The tests are pretty good, and thus they boss me around. Even when I want them to adhere to a new design, like I did today, they still drive me more than I drive them. Brings a slightly new ring to the term "test-driven".
Here's how I originally thought project installation would happen:
Simple, and fine.
Then jnthn showed up (about half a year ago) and told me that that's not how people expect things to happen. So I got a new model, which looks more like this:
Had to drag the tests, kicking and screaming, into this new model, but it was OK. They made sure I didn't botch things up. Things were fine again.
In testing pls with real projects, though, it turned out that the model contained a heinous oversight. See, we only know the dependencies of a project after it's been fetched, but the tests had all their dependencies given up front, and (oops) assumed in some places that they would be.
So I changed that. And everything broke completely.
Why? Because I had used test-driven development, and the code was shaped after the tests. So the code also relied heavily on the faulty assumption. Specifically, the counter-measures for cyclic dependencies seem to have assumed a static model of dependencies, and the cycle-detecting code seemed to end up doing infinite recursive nastiness when it didn't.
Making all of it work meant taking a good look at code that was perfect under the old model, trying to understand why it was less-than-perfect under the new. Ow; brain hurt.
The tests, and judicious debug output, helped me through it. Here's the commit. Now my model looks like this:
And it seems to work. Both the tests and I are happy.
2 years ago today, the Royal navy engaged a bunch of Somalian pirates.
It is only now that I truly realize what an impact Johnny Depp has had on my view of pirates. And the concept "piracy" is hopelessly hijacked by the likes of RIAA. ☹
â¦
Further discussions with patrickas++ today. I'm not getting any code written, but feeling productive nonetheless.
I started by going through all the traversals that the compiler makes. Just explaining it exposed a number of potentially weak spots in the design. The problem with closures that we're having is an interesting one; expressions are traversed bottom-up, but the knowledge about which blocks should be immediate needs to trickle top-down. We're still mulling over that one.
Also, we've made initial plans to create an abstract syntax tree format for Yapsi. Maybe it'll help us solve the current problem we're having with closures; then again, maybe not. There are other potential benefits with having a format sitting between the parse tree and SIC, too. Most of these will become evident as we progress with Yapsi design. The syntax tree format hasn't been nailed down yet, but patrickas and I have decided to name it FUTURE, for reasons which I hope are obvious.
Originally not much more than an April Fool's joke, Yapsi seems to be doing quite well.
2 years ago today, the Royal navy engaged a bunch of Somalian pirates.
It is only now that I truly realize what an impact Johnny Depp has had on my view of pirates. And the concept "piracy" is hopelessly hijacked by the likes of RIAA. ☹
â¦
Further discussions with patrickas++ today. I'm not getting any code written, but feeling productive nonetheless.
I started by going through all the traversals that the compiler makes. Just explaining it exposed a number of potentially weak spots in the design. The problem with closures that we're having is an interesting one; expressions are traversed bottom-up, but the knowledge about which blocks should be immediate needs to trickle top-down. We're still mulling over that one.
Also, we've made initial plans to create an abstract syntax tree format for Yapsi. Maybe it'll help us solve the current problem we're having with closures; then again, maybe not. There are other potential benefits with having a format sitting between the parse tree and SIC, too. Most of these will become evident as we progress with Yapsi design. The syntax tree format hasn't been nailed down yet, but patrickas and I have decided to name it FUTURE, for reasons which I hope are obvious.
Originally not much more than an April Fool's joke, Yapsi seems to be doing quite well.
3 years ago today at the Ibero-American Summit, the Spanish king told the Venezuelan president to shut up. Ok, this takes some explaining.
Of course, it's almost a diplomatic incident when one head of state asks another, in front of cameras and everything, to shut up. Even if the one getting rebuked is acting like a spoiled brat.
As a phrase, "¿Por qué no te callas?" gained a bit of a following.
Here's a video of the event itself. Here's BBC News' take on it. And here's Know Your Meme's.
â¦
Today I meant to dig into the book and give it a good read-through, taking notes. I had it open in one of my browsers, and everything. Everything was just right. Conditions were perfect.
But then I managed to distract myself. patrickas++ showed up with an enthralling piece of SIC that he had hand-translated from a short piece of Perl 6 I had mentioned the other week. The Perl 6 is an example of closures, something that Yapsi doesn't implement fully yet.
We both tried running the SIC code (bypassing the compiler). It worked! I guess this surprised us both. That means Yapsi has the requisite support for basic closures, it's just that the compiler can't generate the SIC that the runtime already runs fine.
So why not just add the functionality to the compiler? Ah, but there's the rub. And that's where the rest of my time tonight went.
Yapsi has gone through one rewrite already. The first version of the compiler used multimethod dispatch to traverse the parse tree top-down and generate the SIC. Result: lots of small methods, all unaware of each other.
The rewrite went in the other direction: it separates the traversal logic into subs (traverse-top-down and traverse-bottom-up), and then does all the specific SIC generation in callbacks to those subs. Result: one big callback with a long if ... elsif ... else comb.
The new code looks scarier and messier than the old code, but it's also more expressive and, in a way, more self-aware. It brought Yapsi further than the old code would. That said, I don't much like the way it looks, and I hope that the compactness and obscurity it currently embodies is a passing phase.
Anyway — and this is why it's not trivial to add closures — there are recursive traversal calls inside the big callback, and those calls carry a @skip argument, telling the traversal logic which nodes to ignore. Among those nodes are block nodes (probably because nested blocks would otherwise end up in each other's generated code). But this blanket ignoring is unfortunate when we start wanting to pay attention to closures in the code.
...something like that. It's not all clear to me yet. I'm hoping that by cramming all of the gory details into my head and then going to sleep, I might wake up tomorrow with a nice solution in my head. Preferably neatly indented.
Hej!
I'm trying to learn Norwegian these days, which is eating many of my round tuits that were previously available for Perl 6 hacking.
On the other hand I'm using Perl 6 to help me learning; I've written PVC, the Perl 6 Vocabulary Coach. It uses a word list, and keeps track of how many times in a row I entered the Norwegian translation of a word correctly.
The better I know a word, the less frequently I'm being asked for it. Not quite the same as spaced repetition learning, but still it feels quite well.
143 years ago today, the Tokugawa Shogunate ended.
I wanted to know a bit more about this Toku guy.
At this point, I realized I hardly knew what a shogun is.
So there you have it. I'm a little curious as to why Toku resigned, and the answer seems to be somewhere in all those history articles; but my background in Japanese history is too bad to separate signal from noise.
â¦
I couldn't focus on anything substantial to do today, so I found the undef warning in Druid, and then fixed a test failure I'd accidentally introduced into the .trans method the other day.
Oh, and yesterday when I was fixing up Druid, I noticed that Rakudo had regressed on &elems (the sub form, not the method form). So today I added it back.
That sums up my activity for the day. Nothing big, just a lot of nibbling here and there.
63 years ago today, a Nazi exhibition called "Der ewige Jude" ("The Eternal Jew") was opened. I can't find any information on Wikipedia about the exhibition, but there's a "documentary" with the same name.
The way I pick these subjects, by the way, is to wander Wikipedia semi-aimlessly until I find something that catches my eye. Today it was this; anti-Jewish propaganda from before WWII. I realize that's not a very cheerful subject, but it was what caught my eye (and I do think that it's an important piece of history to be aware of).
If you are angered or offended by my choice of historical subject today, please let me know. I will also provide a full refund at the exit.
For those of you who also think it's an important piece of history to be aware of, and who are interested in learning more: here's a set of still images from the movie, and here's the movie itself.
â¦
Today I realized some long-standing plans to get Druid to work.
It's been one of the victims of serious improvements to Rakudo (which, as you'll recall, got a new grammar engine going from alpha to ng). Druid was designed with an eye to OO class hierarchies, and even had a few regexes as public attributes. That trick no longer worked.
Well, it did work, if one but added our scoping to those attributes. Except that the regexes insisted on calling each other, and that didn't work. Actually, never mind. The details are complicated and don't matter. I fixed it. Here's the fix.
Current status: all the tests run again! There's one undefinedness warning in there somewhere waiting to be explored. Now that we have Working Code and Passing Tests, I'm sure there are a number of refactorings that could be made, including (but not limited to) the tip Tene++ gave me.
As an extra bonus, the druid script runs! At least to the point where it draws the game board. And it does this with a slow kind of grace. I don't remember it being that slow before. Actually, it looks unacceptably slow.
Tomorrow, I might get around to figuring out why.
60 years ago today during a particularly brisk gale, the Tacoma Narrows Bridge collapsed, thereby winning everlasting recognition in the pages of engineering history.
A newspaper editor, Leonard Coatsworth, was the last person to drive on the bridge. He describes it like this:
Just as I drove past the towers, the bridge began to sway violently from side to side. Before I realized it, the tilt became so violent that I lost control of the car...I jammed on the brakes and got out, only to be thrown onto my face against the curb...Around me I could hear concrete cracking...The car itself began to slide from side to side of the roadway.
On hands and knees most of the time, I crawled 500 yards (460 m) or more to the towers...My breath was coming in gasps; my knees were raw and bleeding, my hands bruised and swollen from gripping the concrete curb...Toward the last, I risked rising to my feet and running a few yards at a time...Safely back at the toll plaza, I saw the bridge in its final collapse and saw my car plunge into the Narrows.
There was one casualty.
And now, if you've never seen concrete wobble before, I'd like you to watch an eerie Youtube clip of the collapse, in recognition of all we've learned about suspension bridges since 1940, and in memory of Tubby the dog.
â¦
Those who tuned in yesterday might recall that I had realized that Str.trans was really, really slow, and that I theorized that it could easily be sped up a bit. Today I put my tuits where my mouth is, and rewrote the Str.trans method to see if it would get any faster.
While preparing the patch, I collected some speed statistics. Using yesterday's blog post (~5kB) as input, and the same arguments as in the .trans call in html_escape, I called Str.trans 10 times and had a script register the time each invocation took. I did this before and after applying the patch.
My patch makes Str.trans about 500% faster than the old version.
As discussed on IRC, the timings of the old algorithm actually get worse over time. This is probably due to a buildup of objects; the old algorithm creates a lot of small strings, which it then joins together. They should be GC'd, but maybe they're not for some reason. The little pink ball at the top is an outlier, a run that was hit particularly hard for some reason, and took 24 seconds. (24 seconds! For 5 kilobytes of text! On a modern computer!)
I wrote yesterday "Now I understand a little better why it is so slow." What I was referring to was that the old algorithm basically works its way forward character by character. Its run time grows proportionally with the length of the string. Maybe that explains why I was the first one to suffer from its abysmal performance: I might be the first one to put longish strings into Str.trans.
What my algorithm does is keep a hash with the next index of each substring to be substituted, and then pick the smallest one through each iteration. (Or, more informally, "skip the boring parts".) This makes the number of iterations through the main loop proportional to the number of substitutions actually made.
Oh, and my code is shorter, too.
Rakudo is often claimed to be slow, and with good reason. However, I hadn't previously suspected that some of the slowness was due to suboptimal algorithms. Now I feel almost morally compelled to read the src/core directory in search for similar spots where we can make quick speed gains just by doing things in a slightly more clever way.
By the way: since Rakudo takes a while to build, I decided to monkey-patch in my version of Str.trans rather than change it in-place and then rebuilding Rakudo each time. This sped me up significantly; all I needed to recompile each time was my small script with an augment class Str in it. I also threw in the relevant spectests, so that they ran after my benchmarking. Nice way to work; gotta remember that.
Oh, and I've put back my html-escape sub into psyde, now that it's not ridiculously slow anymore. So if you're reading this post through RSS, the &'s <'s and >'s in this post have been escaped (efficiently!) with Perl 6.
38 years ago today, the UN passed a resolution to condemn apartheid and to call for a break-off of trade and diplomatic relations in South Africa.
The Wikipedia article about South Africa apartheid is hard to summarize. I chose this paragraph as an example:
The full text of the resolution is available on Wikisource. Its language is formal and dry, but seeing the italicized action words (first participles, then present-tense verbs) at least gives a sense that somewhere in an unjust world, there's a dedicated group of people proposing ways to attenuate the injustice.
â¦
So. With the build time down to about 4 minutes — 4m16s when I ran it a while ago — I already felt a lot better than when it was about 8 minutes. But there was still a lingering, nagging suspicion. It shouldn't take that long; not even for Rakudo. Something was taking a whole lot of time.
Yesterday's post was about pleasing the eye with your code. Today's post is about abandoning your high ideals if it turns out they weigh you down.
Here's today's offending piece of code:
sub html_escape {
# RAKUDO: Need this unnecessary prefix:<~> to make it work
return (~$^original).trans:
['&', '<', '>' ]
=> ['&', '<', '>']
;
}
I say "offending", although there is really nothing wrong with the code as such. In fact, it's brilliant as far as fitness-for-purpose is concerned. If you've ever tried to do the above as a series of regex substitutions, or even (shudder) as a loop doing .substr twiddling, you know what I'm talking about. It's easy to get wrong, especially since the character & sits on both sides of the conversion arrow, so it's possible to accidentally replace already-replaced parts of a string. (An excellent argument against doing such things in-place.) .trans just sweeps these difficulties under the (right sort of) carpet, and gets the job done.
I'm also not referring to the # RAKUDO: comment there, although that one is annoying, in much the same way as flies are annoying. (I just realized I hadn't filed a rakudobug for that one, so I did.)
No, the problem with this subroutine is, unfortunately, that it's written in Perl 6. ☹ At least, that's a problem with current Rakudo. At least, that's what I suspected.
To test this hypothesis, I replaced the html_escape subroutine with a simple shell-out to sed:
sed 's!&!\&!g;
s!<!\<!g;
s!>!\>!g'
Note that this is an exact translation of html_escape, but now we're using regexes. It's important to do the & translation first, for the reason stated above.
Ran it again. 1m22s.
o.O
Wh... I mean. Wha...
I went to look at the source code for Str.trans. It is here. Now I understand a little better why it is so slow. I think making it faster would be an interesting small task for someone, and making it really efficient would make for an interesting medium-sized task.
I love Perl 6. I want to see it succeed. But I've gotten used to pacing myself in various ways. This whole new blog is the result of a paring-down of expectations: it's too early to build ambitious servware applications on top of Rakudo, but it's just the right time to start building static web sites. It's easier to achieve your goals if you make the goals realistic in the first place.
Same thing with html_escape. One day, I'll be able to write the clear, short Perl 6 version and have it run reasonably fast. But that day is not today. Today I'm shelling out to sed, and I'm not too bothered by that. It saves me three minutes with every build run.
Pragmatism trumps purity.
454 years ago today, the (second) Battle of Panipat was joined in northern India. The Wikipedia entry could use some editing, but here's the interesting strategic part of the battle:
Some nice things about this: (1) they were using elephants in battle, that's got to look impressive; (2) one of the combatants is named Akbar. But he's on the winning side, so it does not make much sense for him to yell "it's a trap!". Neither was he a admiral; he was an emperor.
â¦
Instead of taking eight minutes to munch generate stuff for my blog, my static page generator now takes slightly above three minutes.
The change I made is adding in this subroutine, which returns True if a given $target file doesn't exist, or if it is older than its corresponding $source file.
sub nonexistent-or-older($target, :than($source)!) {
return $target.IO !~~ :e
|| $target.IO.changed before $source.IO.changed;
}
I especially like the last line. I read it out loud as "or target IO changed before source IO changed", which is pretty much what the code is doing. I took a teeny bit of a liberty in using before rather than <, because it reads better. Generally I don't like that kind of catering to English-like code, but here it seemed too good to resist. Note that I kept || in favor of or — even I have my limits.
Anyway, it turned out that the subroutine was generalized and lost its pretty perfectly-readable last line; some files like index.html and feed.atom are composed from several source files, and they all need to be checked against the target. So the final version I ended up with was this:
sub nonexistent-or-older($target, :than(@sources)!) {
return $target.IO !~~ :e
|| any map { $target.IO.changed before $_.IO.changed }, @sources;
}
[Addendum: sorear++ points out that I should be writing the above as $target.IO.changed before any(@sources).IO.changed. Clearly that's what the above code wants to look like, platonically speaking. And we're back at extreme readability. I have much to learn still about the wonders of Perl 6.]
Much of the work that remains with psyde is pushing calls such as this into a data structure so that whatever code the user produces, things like this will be called automatically. Right now I do it manually, and my psyde script is ~200 lines long. When everything is stacked neatly into an API, I might get away with as little as ~40. Maybe less.
677 years ago today, the river Arno overflowed the city Florence in Italy. Florentine chronicler Giovanni Villani was there to report it.
...but there is no more information about the big flood in 1333. Only a general hint in the Florence article:
There's a more recent flood, in 1966, that's described in detail. But I can find no more than a general blurb about the flood in the 14th century. Even in Giovanni's source material, linked from his Wikipedia article, I draw a blank.
â¦
This will be kind of a non-post, because though today's explorations have given me reportable results, I'm too tired to sensibly summarize them. So that'll have to wait until tomorrow.
Suffice it to say, perhaps, that psyde is already benefitting from the improvements I promised to install yesterday, and — if my suspicions are correct — will become even faster after some judicious tuning.
Good night.
677 years ago today, the river Arno overflowed the city Florence in Italy. Florentine chronicler Giovanni Villani was there to report it.
...but there is no more information about the big flood in 1333. Only a general hint in the Florence article:
There's a more recent flood, in 1966, that's described in detail. But I can find no more than a general blurb about the flood in the 14th century. Even in Giovanni's source material, linked from his Wikipedia article, I draw a blank.
â¦
This will be kind of a non-post, because though today's explorations have given me reportable results, I'm too tired to sensibly summarize them. So that'll have to wait until tomorrow.
Suffice it to say, perhaps, that psyde is already benefitting from the improvements I promised to install yesterday, and — if my suspicions are correct — will become even faster after some judicious tuning.
Good night.
75 years ago today, the Nationalist government in the Republic of China instituted a fiat currency reform, immediately stabilizing prices and also raising revenues for the government.
And then the war against Japan happened, unfortunately. And then the Civil war, which by the way never officially ended. So here we are, half a century later, unsure just how many Chinas there are.
â¦
I'm in Austria, severely distracted by the beauty of the Vienna. For those of you who have been following my relationship with Austria over the years, I'll just add that I'm not here in capacity of author of proto, I'm here because we're attending the premiere of a documentary in which my mother figures as one of the key characters. Way to go, mom!
Today I toyed around with the idea of making psyde more efficient. I'll step back a few steps to explain what this means.
psyde is the name of the application that generates static HTML for my blog. Right now, all it can do is generate my blog; when I'm done generalizing it, it's supposed to be capable of generating any static web page one tells it to.
Also — and this will probably sound a bit insane — every time I run psyde, it nukes my current local copy of the static web site, and generates a brand new one. That takes on the order of seven minutes. Now, most pages won't change from one such run to the next. psyde currently doesn't care; it just nukes it all and generates it from scratch. Not good.
So what would be a better option? Well, something akin to the ways Makefiles work would be neat. And in fact, this is exactly the problem Hakyll solves, the piece of software psyde is based on. Hakyll uses Kleisli arrows (a data structure similar to monads) to keep track of dependencies and what needs to be updated. I meant to do something similar in psyde, I just haven't gotten around to it yet.
But today I've toyed a bit with the idea, locally. Maybe tomorrow I'll even have enough working code to attempt to put it in psyde itself. It'll be interesting to see how much the run time goes down.
248 years ago today, the Rt. Hon. Dawsonne Drake became governor of Manila in the Philippines. Ruling for 20 years, he was the only British governor of Manila during the British occupation of the Seven Years' War.
At this point, I realized I knew far too little about the Seven Years' War, so I went and read a bit about that too. Take-away points: it was so big that Winston Churchill called it the first "world war". Also, the Last of The Mohicans takes place during this war.
Anyway, governor Drake. Wikipedia writes about his inglorious return from Manila:
I hope they did the thing where they rip the medals off his chest.
Ok, so he got to keep his job and the loot? Sounds almost like the CEOs of today...
â¦
I decided to start simple today, and see if pls still passes its tests. It didn't.
Turns out Rakudo has bitrotted, and moritz++ had fixed the json repo. pls carries its own copy of the exact same module files (for bootstrapping purposes), and so I had to apply moritz' commit to pls as well.
A nice thing was that git allowed me to git format-patch moritz' commit and then git am it to the pls branch of proto. It wasn't a big commit at all, but it's nice to know that that works (as long as the directory structures match).
Anyway, now pls passes all tests again. \o/
I'll stop here for today, but I'll note what the next step for pls is: the tests need a slight refactoring, because they currently pretend that a non-fetched project knows about its dependencies. That's not wrong, because the dependency metadata (contained in a deps.proto file, for example) is fetched along with the rest of a project, so we can't know dependences until after fetching. This test file is So Full Of Wrong for that reason. Last time I tried to fix it though, I didn't manage to mend things after I'd broken them.
Maybe next time I'll be more lucky. We'll see!
Of the six goals in my last post, four of them now work well. Here’s the latest PDF my code has produced. I’ve added three new tunes to it: another of my tunes, “Tali Foster’sâ€, a quick snippet from our ceili band’s notes from a few years back, and a double from the repertoire of Rufus Guinchard, “Sydney Pittman’sâ€. Together they demonstrate time signatures (6/8 and a single bar of 9/8), key changes, broken rhythms, and of course more than one tune on a page. (Note that 1st and 2nd endings are still unimplemented, and look very wrong in “Tali Foster’sâ€.)
I have to say that this PDF really impressed me with Lilypond’s output. It’s hard to put my finger on it, but something about having all four tunes together like that on the page looks really good, IMO. I’m getting excited about the prospect of being able to produce entire books of tunes with this tool.
In terms of Perl 6, this last nine days of work has been very straightforward. I did whine enough about the undeclared variable error message bug to convince pmichaud++ to fix it. I’d worried a lot about nested levels of Contexts — the key, time signature, etc — but I realized as I looked at it there are no nested levels, all changes affect the rest of the piece (until there is another change). I refactored to change several of the subs into methods, and made the context an attribute of their class. I’ve set things up so that Context objects are read-only, and you just create a new one when you need to change the context of the tune. So far this seems to work well.
I guess at this point I really need to implement 1st and 2nd endings, and then push on by throwing more tunes at it to see where they don’t work.
Update: Just in case you wanted to see what “Tali Foster’s†was supposed to look like with proper repeats, here’s the latest PDF. That is to say, first and second endings now work, at least in this one case. I’ve also realized that while key signatures work at the moment, accidentals don’t properly modify the key signature until the end of the bar — so that’s another thing that needs doing.
It is with a contagious joviality that I announce on behalf of the Yapsi development team the October... hm... November 2010 release of Yapsi, a Perl 6 compiler written in Perl 6.
You can download it here.
Yapsi is implemented in Perl 6. It thus requires a Perl 6 implementation to build and run. This release of Yapsi has been confirmed to work on all releases of Rakudo Star to date.
Yapsi is an "official and complete" implementation of Perl 6. The fact that it's official means that its author has claimed that it's official for over half a year now, and not once been contradicted. Hence, it must be so. The fact that it's complete means that the code has the same volume (in cubic centimeters) as its packaging.
This month's release brings you mostly bug fixes... BUT it also comes with compatibility for the long-awaited Tardis debugger (sold separately) which allows ordinary mortals to jump backwards in time!
$ export PERL6LIB=path/to/yapsi/lib:path/to/tardis/lib
$ bin/tardis -e 'my $a = 42; my $b; { $b = 5 }'
# Compiling...
# Running...
# Finished. Ticks: 0..3
> go 1
$a = 42
$b = Any()
> go 3
$a = 42
$b = 5
> go 1
$a = 42
$b = Any()
> ^C
For a complete list of changes, see doc/ChangeLog.
Quite a lot of features are within reach of people who are interested in
hacking on Yapsi. See the doc/LOLHALP file for a list of 'em.
Yapsi consists of a compiler and a runtime. The compiler generates instruction code which the runtime then interprets. (If this sounds tricky, think of an assembly line where little toy soldiers are constructing tiny toy-soldier-sized assembly lines, and so on all the way down to the scale of quantum foam. Now THAT's tricky!) The format used to represent the instructions in Yapsi are called SIC. Being of questionable merit from the beginning, SIC changes all the time, except for the property that it changes all the time, which has tended to be quite predictable. Even that, however, may change over time. Anyway, don't expect the SIC of next month to look anything like the SIC of this month. It might, but don't expect it.
An overarching goal for making a Perl 6 compiler-and-runtime is to use it as a server for various other projects, which hook in at different steps:
Another overarching goal is to optimize for fun while learning about parsers, compilers, and runtimes. So far that goal is on track, and more.
Have the appropriate amount of fun!
37 years ago today, a coup was carried out against Vietnam's first president, Ngô Äình Diệm.
Diệm had lost his US backing after religious protests in Vietnam, and the US had given "secret assurances to the generals that the U.S. would not interfere" with the coup.
â¦
Today, I dug my teeth into Yapsi and — quite unexpectedly, actually — Tardis.
For those of you who don't recall what Tardis is: it's a debugger with the special twist that you'll be able to jump back and forth in the program, not just farwards.
Fixing things up today, I felt a bit like David Tennant, running around inside the Tardis, banging on things with a hammer. Things work now, but they're neither elegant nor particularly user-friendly. Just like the real Tardis, I guess. 哈哈
Anyway, things have been going smoothly, with a few commits in both the Yapsi and Tardis repos. I had no unforseen blockages that stopped my progress today. And now, it's time to go make a Yapsi release with some Tardis on the side.
Starting tomorrow — actually later today — I will make a semi-short post each day. You have been warned.
Each such post will consist of two parts: one with a retrospective into an event or phenomenon taking place on that particular day in November sometime in history; the other with some modest update representing around half an hour of work on something Perl 6-related. If this all sounds eerily familiar, it might be because this is the third year I do this. It also happens to be the last.
What do I pledge to do in the coming month? This:
master, as opposed to the alpha branch.master.pls, such that it installs the whole ecosystem.So that's it. Mostly my work will focus on November (hence the choice of month, after all), or on webby stuff.
Lately I feel I've lost contact with Actual Real Codingâ„¢, sort of sniffing at the outskirts of it, but not sitting down and practising it much. I see this as my chance to jump (or fall, or trip) back into the fray a bit. Looking forward to that.
Go go go!
A couple of weekends ago, I wrote a post on the difficulties I was having with the multiple dispatch specification as it stood then. Early this week, while I was visiting some good friends in the UK, TimToady++ made a spec commit aimed at addressing the issues I raised. A first scan through it on my phone looked promising; a closer reading and a further conversation once I got back home left me feeling like we now had something very implementable indeed. And so I set this weekend aside for trying to get an implementation together in the .Net implementation of NQP.
On Friday evening, I warmed up for the weekend’s hacking with a couple of good beers, finishing return exceptions so that one could use the “return†keyword and getting closures basically working. Kinda fun way to spend an evening, and unblocked some other things.
Things started out well. After a tiny preparatory commit to let a routine potentially act as a dispatcher, I then worked to do the minimal set of changes to get from the previous attempt I made at multi-dispatch (where I started to discover the spec problems) to something shaped much, much more like the model described in the updated specification. I lost 5 minutes trying to work out why my changes hadn’t built, only to find they had built, it was just that I’d managed to do the model switch without causing any test regressions at all, on the first attempt. Sometimes I’m almost competent…
From there, I was able to quickly get the case of a nested lexical scope having its own instantiation of the proto that presides over its candidate list implemented and add support for the * syntax for indicating in the proto where to call the multi-dispatcher. At that point, the NQPSetting could start to look a bit nicer.
As a stress test, I decided to see what happened if you wrote a proto that, instead of entering the multi-dispatch, returned a closure that would do so.
proto wow($x) {
say("in proto");
return { say("in closure"); {*} }
}
multi wow($x) { say($x) }
my $c := wow("in multi");
say("back from proto");
$c();
Which, after fixing a really silly thinko, worked:
in proto back from proto in closure in multi
I think that almost scores full marks for abuse of a proto (though somebody will probably find a way to do better… :-)). But most happily, it tested multiple dispatch, closures and return all in one go – which at that point I’d managed to get in place in the space of the previous 24 hours. I took a little Ukrainian vodka, relaxed for a bit and got some sleep.
Having got the multi-subs done, I figured I could try my hand at multi-methods. This presented me with the slight issue that I hadn’t actually got enough of NQP’s ClassHOW filled out yet to have a sufficiently advanced class implementation to even ponder this. After a couple of small commits to remove blockers, I want ahead and did so and…bingo. Inheritance, and attributes too. Whee. (Note from this patch, how I’m actually able to implement the semantics of NQPs classes by writing in…NQP. This code is actually what now runs and provides classes in NQP on .Net.)
Multi-methods turned out to be something that I could mostly implement just by adding to NQPClassHOW. First I implemented the simple case where the proto is in the class itself so we don’t have to go off looking through the MRO for it and can just start adding candidates. With that working, I implemented the more complex case where we go and find a proto and instantiate it. It used the exact same underlying runtime operation that the nested lexical scopes did – a unification I’d expected to be able to make and was comforted to trivially achieve.
The upshot of this weekend’s hacking is that I now have a running reference implementation that delivers the proto semantics in the current specification – or at least, the key ones such that they affect dispatch. I’m happy enough that I know how, in the future, to achieve wrapping and deferral within this model, and have most of the pieces in my head for how to handle proto inlining in the trivial onlystar-body case too (which will save having to create a callframe and so forth for most protos). Next steps for the multi-dispatch work will include porting all of this over to the Parrot implementation of NQP.
As an aside, I’m also extremely happy with the way NQP on .Net is coming together. What started out as a way for me to play around with how to factor the new meta-model is shaping up to be a great prototyping ground for other features. But more than that, I think it’s got real promise to grow up first into a fully bootstrapped NQP implementation – delivering NQP and the compiler toolkit to the .Net platform – and then to go on to form the foundation of a Rakudo backend for .Net too.
I'm mostly happy about the Perl 6 specification. Not all of you may know what it is, so here's a short summary: it consists of several numbered synopses, each of which describes a certain part of Perl 6. Some of the synopses sprang into existence as summaries of correspondingly-numbered "apocalypses" — those are the original Perl 6 design documents written by Larry Wall as a summary of the conclusions of reading through the RFCs. The apocalypses are way out-of-date nowadays; they're littered with updates that alert the reader to how syntax and other things are nowadays, but many of these updates are also out of date. The real specification is synopses; that's where the language is defined.
That's the specification. Several synopses, each of them about some
topic. Sometimes STD.pm6 is thrown in as being part of the specification,
too, especially as it still contains ideas that haven't made it into the
synopses yet.
As different implementations (mostly Rakudo nowadays) catch up with the specification and actually implement it, we become increasingly confident that it actually works as specified. That's code for "we go in and change it a lot until the specification agrees with the implementation". This is a beneficial process, and it's part of why I really like Perl 6: we're not afraid to adjust our dreams to conform with reality.
This back-and-forth between implementations and specification is of course what the in-house term "whirlpool development" is all about. It's a sort of antithetical stance to "waterfall development", this notion that work should proceed in discrete stages, from analysis via design to implementation and then testing. The whirlpool model maintains that later stages (like implementation and testing) affect earlier stages (like analysis and design). And boy, is that ever true!
In fact, this has sunk so deep into our mental model of the project, that we don't really trust a synopsis until it's been implemented. The synopses are the rough consensus that we've all agreed upon so far, but until there's running code too, the synopsis is assumed to come with a big grain of NaCl.
The terminology that seems to emerge for this is that a synopsis becomes increasingly frozen over time, but before it reaches that point through actual empirical implementing, it's slushy or even liquid. Today, we have a few frozen bits of spec, lots of slushy bits, and a few liquid bits. Slushy is good; it means that we have some backing from implementations, but the spec and the implementations don't quite agree yet. I like that part, and since I'm an early adopter by temperament, I'm fully prepared to work under such conditions.
Concrete example: Larry made some changes to multi semantics; Jonathan got visibly worried, but couldn't quite put his finger on why. Recently he put his worries into words, which unsurprisingly were backed up by implementation concerns. Larry obliged and refined the semantics to address those concerns. The implementations drive the specification.
That last example is also a suitable instance of another metaphor: various parts of the spec seem to undergo some kind of stochastic hill-climbing in a hypothetical space. Along the way, it might reach several local maxima, in which some of the concerns have been addressed, but there is still gnashing of teeth over others. Usually, when we reach a global maximum, everybody just knows: this is it. (Well, at least to the extent that hill-climbing guarantees that the maximum really is global.)
All this is emphasized in the Perl 6 motto "Second System Syndrome Done Right". Redesigning things on the basis of empirical feedback. And it's pretty good.
There are also a couple of pieces of spec where I think we haven't found the global maximum yet, or even a local one. That's what I really set out to write about today. I feel a bit like MJD now; what you read so far was only an introduction of sorts.
Let's split things into two parts: slight unease and abstraction astronautism. The former are parts where I believe we're on the right track but not quite there yet. The latter are parts that I believe were too ambitiously designed from the beginning, and that are probably better thrown out wholesale. The former requires out-of-the-box thinking; the latter would need something more like out-of-the-box courage.
The angle brackets <foo bar baz> provide the new, shorter qw syntax in
Perl 6: the example gives you the list ('foo', 'bar', 'baz'). But people
keep using this construct, essentially a string quoting mechanism,
expecting to get numbers out when they write a number. Due to this, the
spec contains special concessions for any items in the list that look
like numbers; each such item will be stored as "an object with both a string
and a numeric nature". This unnerves me; I'd be happier if the construct
always returned pure strings, that I'd then have to convert manually. I
cannot quite put my finger on why I think this is bad and could create
problems down the road; I just do.
A related, but almost the opposite kind of problem: if I create a
MAIN sub with an Int parameter, there's no command-line argument that
would match that, since all the arguments coming in from the command line
are, by definition, strings. I really like that we use signatures and multi
subs to match command-line arguments; that just feels so right. But it feels
like a waste to never be able to match an Int like that. Whereas I
consider the previous situation to have too much magic in it, here I feel
there's too little. I'm not at all sure that "an object with both a string
and numeric nature" would be a good solution here either; I'd prefer
something less icky that still solves the problem.
I used to be slightly uneasy about the way the empty list () worked.
Now, after some spec
changes,
I'm mostly relieved. Now sorear++ is uneasy instead, for (he says) reasons
having to do with optimization.
Most of synopsis 5 (about regexes and
rules) is really frozen by now, thanks to some excellent implementations.
(The really slushy phase was sometime in 2006, if I recall correctly.)
But last time we talked about the <?after ...> assertion, it seemed to me
that there were more questions about it than there were answers in the
synopsis. "Lookbehind" assertions, according to S05, work by "reversing the
syntax tree and looking for things in the opposite order going to the left".
Of course, that only works for syntax trees without captures and other
side effects, so the spec also says "It is illegal to do lookbehind on a
pattern that cannot be reversed." Or, in other words, we haven't reached the
point yet where we need to do lookbehind with those more tricky syntax
trees. Fair enough.
Let's start with a sunshine story: the Temporal
spec. It used to be really
bad, but after work that I started and Kodi++ and supernovus++ finished, we now have a sane
spec and a full Rakudo implementation. The secret sauce? Throw everything
out and just copy someone else. In this case, we mostly copied the successful
DateTime CPAN module, with some helpful experience-based tweaks suggested
by autarch++, its original author. Essential to this type of wholesale
rewrite is that it be done by one or at most two people. The many-cooks
syndrome is usually what got us into trouble in the first place.
The IO spec is a mess. Most of it
is completely unimplemented. Some parts of it, if I recall correctly, are
unimplementable, or would be a big mistake to implement as spec'd. It's my
hope that this synopsis will follow the example of the Temporal one,
and that someone would pick it up and just rewrite it from scratch. As an
example of the atrocities perpetrated in this area, someone once set out to
design a Tree core data type that would encompass such diverse structures
as file system hierarchies and XML DOM structures. This, in my view,
epitomizes abstraction astronautism. The effort with the Tree
specification seems to have fallen on its own absurdity.
Sometimes I question whether Perl 6 should really embrace and extend Unix
command-line options in the way S19
says it should. What really
seems over-the-top to me is the way one can specify namespaced options by
means of a special type of "parenthesizing" flag — think SGML tags.
This kvetch could easily have fallen under the "slight unease" section, but
I sincerely believe that some of the more ambitious parts of S19 should just
go away. Having it conform better with the MAIN spec would be nice, on the
other hand.
A third category would be "areas where I wish there were more spec, period". But searching for those is the objective of being all over the place with Rakudo, writing different application code, trying new and exciting things. Such discussion take place all the time on IRC, over pieces of code, or on the p6l list. So I don't have a pent-up need to blog about those.
So many amazing things happened at the Google Summer of Code Mentor summit 2010! I will try to jot a few of them down, before they leave for warmer climates. For those that just want to read all the session notes, you can find them here. Also, if you haven't yet read about how The Perl Foundation and Parrot Foundation fared this summer, you can read about it on the Google Open Source Blog.
It began by arriving a bit early to work with some awesome people on improving the GSoC Mentor Manual by adding a chapter for Organization Admins (there is actually documentation now!) and writing a GSoC Student Manual. This "book sprint" was facilitated by Adam Hyde of FLOSSManuals, and they were written with a completely open source software stack, as well as being released under a Creative Commons license. They are free for anyone to read online and are easily exportable to many formats. Read the Student Manual or the Mentor+Org Admin Manual online now! We even bound 60 copies of the books and handed them out to mentors attending the summit.
Parrot on RTEMS hacking with Chris Johns and Ralf from RTEMS. We used Centos 5 RPMS on Ubuntu 10.04 with rpm2cpio piped to cpio, which was a trick to get around the fact that RTEMS does not have debian packages. It worked remarkably well. I had a cross-compilation environment setup after a few minutes. I think they will be adding these intructions to their wiki. Now that I have the RTEMS toolchain on my netbook, I will be much more productive with regard to Parrot on RTEMS.
Chromatic, Chris Johns and I sat in a room and talked shop about how Parrot and RTEMS can play nicely together. There are still some feature voids on the Parrot side to fill: Parrot calls exit() in various places, which reboots RTEMS i.e. a syntax error reboots the OS. Not Fun. Parrot also needs a C probe to detect RTEMS, which already has a ticket in our bug tracker. A real-time garbage collector will be needed for long-running processes, but for short-lived applications, disabling the GC with the -G command line argument to Parrot will work.
I gave a session with Selena Deckelmann and Bart Massey introducing Troll University, which aims to educate organizations, corporations and open source communities about what motivations and principles trolls use and how to protect against them. We are working on some Trollcasts, so stay tuned!
I also gave a session called Dynamic Language Interoperability, which has been held for the last few years, to my knowledge. The consensus seemed to be that every dynamic language has the same interop problems, and Parrot VM seems to be the only project working hard to solve these complex issues in a general manner. This gives me a lot of hope that people will soon realize that Parrot is full of WIN.
It also came to my attention during the conference that Github hired the student that mentored under them this year to work on libgit2. This is one example of the amazing opportunities that students have after completing a Google Summer of Code. The sky really is the limit. And just in case you think this is an isolated incident, it isn't.
As if writing some books and going the Mentor Summit wasn't enough to totally drain me, I am currently attending the last day of the GIT Together 2010, which is the yearly Git developer and user meetup/unconferencey thing. I have learned so much that I can't even begin to describe it, but if you want to look at session notes, you can find them here.
I’ve been slowly poking away at the ABC module without reporting on it here, as the changes have been pretty straightforward. All of the goals at the end of my First Sheet Music post have now been achieved. Here’s the latest version of “The Star of Rakudo†PDF. If you compare it to the output of my old sheet music generator, you’ll see every musical element except the time signature is now correctly rendered by the combination of the Perl 6 ABC code and Lilypond. (In addition, I’ve also added support for rests and triplets, which are now found in the module’s version of the Star of Rakudo ABC file as an example.)
Where to go from here?
1) I guess I ought to fix the time signature thing. That should be trivial.
2) Support for ABC files whose base note duration is something other than an eighth note. (Right now we’ve just hardcoded things to assume eighth notes are the base unit of time.)
3) Broken rhythms.
4) In-line time signature and key signature changes.
5) Handling more than one ABC tune at a time in the input.
I don’t see any major challenges here, other than finding the time to work on this project!
Update: Later in the same day, I’ve already got #1 and #5 working, but I just realized I left out one important (and possibly tricky) one:
6) Handling first and second endings.
In the Perl 5 world, "core" modules are a big topic. Some people can't (or think they can't) install any modules beyond "core", and suggest to add more useful modules to "core". Others think that we should have fewer core modules, to reduce the maintenance load on the perl 5 porters.
So, what's the deal in Perl 6? which modules will be "core", and who decides? Will we learn from Perl 5's mistakes, whatever you consider them to be?
To answer that question we have to explore what "core" really means. In Perl 5, it's just a module that's included in the "perl" package that's uploaded to CPAN. In Perl 6, there are several possible meanings:
Note that all three concepts known in Perl 5 too - UNIVERSAL is category A, warnings is category B, and there are some modules that are installed by default by ActiveState Perl or Strawberry Perl.
The Perl 6 specification talks about category A core modules in the S32/Setting Library documents. Currently things like DateTime, Date and IO::Socket::INET fall into that category. Maybe someday they will be automatically loaded on demand, to cause less strain for all programs that don't use them.
Category B is currently rather sparse, it only consists of Test.pm (and in the case of Rakudo also Set.pm, but that should eventually become a category A core module). This is where we learned from old mistakes: keeping compilers and distributions separate reduces the maintenance load for the compiler writers. Do you as a user want a module installer? then you should be installing a distribution, not just a compiler
Everybody can start their own distribution, based on any existing compiler. That's a good thing. Whoever maintains a distribution gets to decide what to ship with it. Currently the Rakudo developers both write a compiler (commonly referred to as Rakudo), and a distribution called Rakudo Star. The compiler will continue to ship with very few modules, the distribution will adapt to the need of the users.
I fully expect there to be specialized distributions in the next few years or decades: maybe a sysadmin distribution, a web developer's distribution and so on. Or maybe that specialization will be realized by bundles. We'll see.
Perl 5 also has "dual-life" modules, which are both released to CPAN and are part of the core. Since Perl 6 has almost no type B core modules, there will be almost no need for dual-life modules either. I expect that most distribution makers will rather import modules that are maintained by third parties, rather than maintaining them on their own - kinda like Linux distributions package popular and useful software, but only actively maintain it in rare cases.
The middle challenge was to find the first prime Fibonacci number greater than 227000, add one to it, and then sum the prime numbers which were its factors. Here’s my first implementation:
sub is-prime($a) {
return Bool::True if $a == 2;
! [||] $a <<%%<< (2, 3, *+2 ... * > $a.sqrt);
}
my @fib := (1, 1, *+* ... *);
my $cutoff = 227000;
my $least-prime = 0;
for @fib -> $f {
next if $f <= $cutoff;
next unless is-prime($f);
$least-prime = $f;
last;
}
my $x = $least-prime + 1;
say [+] (2, 3, *+2 ... * > $x.sqrt).grep({ $x %% $^a && is-prime($a) });
Despite what seems like an obvious inefficiency (approximating the prime numbers with the odd numbers), this is pretty snappy, executing in 12.5 seconds.
I was planning to go on and talk about my new Math::Prime module here, but looking at this code, I think it can be expressed rather more nicely with a tweak or two here. Let’s see.
sub is-prime($a) {
return Bool::True if $a == 2;
! [||] $a <<%%<< (2, 3, *+2 ... * > $a.sqrt);
}
my @fib := (1, 1, *+* ... *);
my $cutoff = 227000;
my $least-prime = @fib.first({ $_ > $cutoff && is-prime($_) });
my $x = $least-prime + 1;
say [+] (2, 3, *+2 ... * > $x.sqrt).grep({ $x %% $^a && is-prime($a) });
So that’s what the first method is good for!
I did indeed write Math::Prime just so I could use it here. It’s not a huge change from the previous version, really:
use Math::Prime;
my @fib := (1, 1, *+* ... *);
my $cutoff = 227000;
my $least-prime = @fib.first({ $_ > $cutoff && is-prime($_) });
my $x = $least-prime + 1;
say [+] (primes() ... * > $x.sqrt).grep({ $x %% $^a });
Unfortunately, Math::Prime isn’t optimized yet, and so this version, while a bit nicer, is actually slower than the previous version.
Update: With Moritz’s help I did some optimizations to Math::Prime, and the script using it is now very significantly faster than the others.
For this week's contribution to Perl 6 we ask you to improve the error message that Hyper Operators emit when lists are not of equal length.
(Introduction to this series of challenges)
In Perl 6, operators can be applied to lists; in one version, the lists must be of equal length, in the other form one or more lists are automatically repeated to be of proper length.
# equal length required: say join ', ', (1, 2, 3) >>+<< (10, 20, 30); # output: 11, 22, 33 # auto-extending the right side by # turning the less-then/larger-then signs around: say join ', ', (1, 2, 3) >>+>> 10; # output: 11, 12, 13 # this correctly produces an error message (1, 2, 3) >>+<< (1, 2) # output: Sorry, sides are of uneven length and not dwimmy. # recurses into structures ((1, 2, [3, 4]) >>+<< (10, 10, [20, 20])).perl #output: [11, 12, [23, 24]]
Please submit a patch that improves the error message. A good better error message would be:
# for non-recursive structures: Sorry, structures on both sides of non-dwimmy hyperop are not of same shape left: 3 elements right: 2 elements # for recursive structures: Sorry, structures on both sides of non-dwimmy hyperop are not of same shape left: 3 elements right: 2 elements at nesting level 2
The source code can be found in src/core/metaops.pm, spread out over a few multi subs.
You'll need to introduce another (probably named) parameter or a contextual variable to track the recursion depth.
Update: I have received one submission, worked a bit on it and included it in Rakudo.
Please submit your patch to the [email protected] mailing list (and put [email protected] on CC, because the mailing list sometimes lags quite a bit).
If you have any questions, feel free to send them to the mailing list, or ask on our IRC channel".
It turns out that I’m having to take a slight diversion from meta-model work to spend a while worrying over dispatch – because not dealing with the design issues now will only make for more refactoring and pain later. Further to that, it’s going to have an impact on how bits of the meta-model API should look.
Dispatch is essentially about flow control between chunks of code at the inter-block level. We often talk about dispatch as being what happens when:
However, I’m also wondering if we should consider the case of throwing an exception in with dispatch. It is also a dispatch in a sense – a dispatch to an exception handler. Running the handler may or may not indicate that some kind of stack unwinding is required. But actually getting to the handler is a kind of dispatch.
It may be useful to break dispatches up into groups.
For all complex dispatches, we have the concept of a candidate list. A candidate list is a list of things that we can work out way through calling. In a method dispatch, it’s all the methods through the MRO with a matching name. In a multi-dispatch, it’s all the matching candidates. In the case of wrapping, it’s the various wrappers, down towards the original routine that we wrapped. When we have these candidate lists, we have various operations that we can perform on them.
In some cases, we may end up with three different dispatches active at the same time for what is, from the caller’s perspective, just a method call. This shows up if you call a method (thus we’re in the method dispatcher) that happens to be a multi method (so we’re going to need to use the multi dispatcher) but somebody wrapped the proto (so we’re working our way through the wrapping candidate list too).
So, here’s some of the things I’m pondering.
First up, let’s look at how we obtain and iterate over candidate lists. For all complex dispatchers, in the general case, we have no idea up front whether we’re going to need the whole set of possible candidates or just the first one. (We may one day be able to build sufficient static analysis to know in some cases.) This means that we either need to:
Second, dispatchers are first class from a referential point of view even if not in their implementation (actually, the issues are almost certainly orthogonal). That is, if I write &foo and we have some multi foo in scope then I refer to the whole set of candidates available from the point of view of the current scope. That is, in:
class Drink { }
class Beer is Drink { }
proto foo($x) { {*} }
multi foo(Drink $x) { 42 }
my $c;
{
   multi foo(Beer $x) { 69 }
   $c = &foo;
}
say $c(Beer.new);
We’d expect the output to be 69, not 42.
Similar for $obj.can(‘meth-name’) – it returns something that we should be able to invoke and that will provide access to the whole candidate list. In these two cases, what are we actually getting a reference to? It can’t simply be a code object because we expect deferral to work. Instead, it probably needs to be a “dispatcher instanceâ€, instantiated with (or curried with) the candidate list.
This is not really stated in the Perl 6 specification anywhere though, so far as I can see. It perhaps needs to be, however, because we probably have to make the differentiation in ways that are user visible. For example, I expect that taking a multi and doing .candidates, or getting a method through .^methods, will almost certainly not give you something that will ever work in the case of deferral, since it gets you a code object, not something with a candidate list.
This code object vs dispatcher distinction comes and bites fairly hard in the case of multiple dispatch. A while ago, the specification for multiple dispatch changed quite drastically. I remain disappointed that I wasn’t consulted on the changes by the Perl 6 design team, given I’ve led the way in implementation work on multiple dispatch in Perl 6 in the last couple of years. The essence of the change is that whenever you do a multiple dispatch, you first call a proto, which in turn delegates to the multi-dispatcher.
This was done to help answer questions like “what does it mean to wrap a multiâ€, which was just not possible before. Now it simply means “you wrap the protoâ€. Answers to “what is the arity of a multi†and “what is the signature of a multi†now boil down to the same answer – whatever the proto declares. Further, this means that you have the possibility to do something pre-dispatch and post-dispatch. These are the good consequences, and I don’t disagree with they are improvements. However, there are some less good ones too.
One immediate one – though one I know how to deal with – is that now every single multi-dispatch is, in the unoptimized case, two Perl 6 level invocations. Remember this hits every single operator invocation. Yes, ouch. This means that if we want to implement this scheme without a bad performance hit, we have to implement some basic inlining support.
An issue I don’t want to consider right now – but will probably happen – is a stream of (very justified) complaints from Perl 6 users that “I have to write this proto thing that I never had to before – why the boilerplate?â€
Here’s the one that is making my life much trickier, though. Consider:
class Drink { }
class Beer is Drink { }
proto foo($x) { {*} }
multi foo(Drink $x) { 42 }
my $c;
{
   multi foo(Beer $x) { 69 }
   $c = &foo;
   &foo.wrap(sub ($x) { "lol " ~ callsame; });
}
say $c(Beer.new);
say foo(Beer.new);
I think most folks would agree the output of this should be “lol 69″, followed by “lol 42″. The question is, how do we actually make this work? Consider what’s in $c. It references some &foo from the inner scope. But here’s the problem: what is this &foo?
That is, it needs to be the proto apart from when we do a dispatch. But that’s not where the problems end. Suppose we did manage to delegate all method calls off to the proto apart from .candidates, which obtains the current view of the candidate list. How does the {*} inside the (now wrapped) proto, which actually enters the multi-dispatcher, get hold of this candidate list?
We have a kind of inside-out thing going on here. On the one hand, we’re invoking the proto to do the multiple dispatch, but the proto is actually just calling a wrapper, which wants to have its own candidate list. That is, if you imagine a stack of dispatchers, we have the candidate list for the multi-dispatcher in our hands when we invoke the proto, but it instead wants to push a wrapper dispatcher on the stack to go through the wrapper candidate list. It’s not until we hit the {*} – if we ever do – that we need to take the candidate list from the scope we captured a reference in (or invoked from, in the simple case) and give it to the multi-dispatcher.
And coming up with a robust – let alone performant – implementation of this is what’s currently tying me in knots. I didn’t even talk about closure semantics yet. Consider:
class Drink { }
class Beer is Drink { }
proto foo($x) { * }
my $c;
multi foo(Beer $x) { $c = { nextsame } }
multi foo(Drink $x) { say "badger" }
foo(Beer.new);
$c() for ^4;
For this to work, the current state of the candidate list iteration also needs to be lexically scoped too. We currently make this work in the methods case in Rakudo by making the dispatch state immutable and storing it in a lexical. Anyway, it’s yet another thing to consider into the design.
I guess my overall gripe here is that we’re going from a model that) is easy enough to implement and understand (despite having some real issues for “power usersâ€), to something that seems to have a less obvious clean implementation, makes people write more boilterplate and relies on inlining optimizations to be decently performant. And while the specification offers the implementor a wish list of things that should work, there’s not so much in the way of guidance on how one might actually structure things.
Perl 6 allows us to define classes and roles and grammars, and they all look much the same.
class C {
method foo { ... }
method bar { ... }
}
role R {
method foo { ... }
method bar { ... }
}
grammar G {
regex foo { ... }
token bar { ... }
}
The similarities don't just exist on the surface; all these three OO-style packages get parsed with the same rule (package_declarator) in the Perl6 grammar, and their insides get parsed with the same rule (package_def). The only difference is the metaclass object handed to each one. ("What's a metaclass?" you might ask. If you are, I recommend this post by jnthn++.)
So classes get handed a ClassHOW, roles a RoleHOW, and grammars a GrammarHOW. And that's basically how they differ. In theory, if we wanted to give (Rakudo) Perl 6 a bigger repertoire of OO-style package declarators, all we have to do is add a few rules to the Perl 6 grammar, and add a new SomethingHOW metaclass.
The operative phrase being "in theory". See, no-one has ever done this, and whereas things are designed for people eventually being able to do that. Right now, you're advised to sit back and wait while some foolhardy person gets a thousand small cuts and bruises trying to make it all work without a hitch.
That's where I come in.
This is going to look almost trivial. Well, it is trivial to describe. One day it'll be trivial to get to work in Rakudo as well. Here's the new syntax:
diagram D {
element foo { ... }
element bar { ... }
}
Hm; actually, I have a full-blown example of the diagram DSL, that I might as well show at this point.
class Page {
has Str $.style = 'fill: #333333';
}
class Book {
has Str $.abbreviation;
has Str $.name;
has Str $.author;
has Str $.year;
has Page @.pages;
}
diagram BookLayout {
element LAYOUT { <booktable> on <attribution> }
element booktable { table(<book> from .books, :4columns) }
element book { .abbreviation on table(<page> from .pages, :10columns) }
element page { <rect width="1" height="2" style="{.style}"> }
element attribution {
vline on "SOURCE BOOKS"
on table(<source> from .books, :2columns, :down)
}
element source { .name on "by $_.author, $_.year" }
}
my Book @books = ...;
BookLayout.render(:@books);
The above example was written to produce something very much like the third image over at One book, many readings. (Here's a direct link to the image.) Compare the elements in that image and the elements in the BookLayout diagram, and you'll see the similarities.
The final line, BookLayout.render(:@books);, would spit out a string full of SVG, or some other scene description format. The array @book would contain 12 Book objects corresponding to the 12 books in the image.
Anyway, I thought this was a nice idea. It's certainly the best idea for a non-core extension to Perl 6 that I've been able to think up so far. The syntax of the element blocks is reminiscent of Perl 6 regex syntax, but the process is almost a reverse of the one for regexes: whereas a grammar does a complicated dance over a string of text and eventually produces an object hierarchy (a parse tree), a diagram does a complicated dance over an object hierarchy (in this case, a set of Books with sets of Pages) and produces an image.
Did I get Rakudo to parse my new syntax? Yes, I did. Here's how to do it.
I wanted to add the package declarator diagram and hook it up with a metaclass DiagramHOW. So I added these rules to src/Perl6/Grammar.pm:
%*HOW<module> := 'none';
%*HOW<class> := 'ClassHOW';
%*HOW<grammar> := 'GrammarHOW';
+ %*HOW<diagram> := 'DiagramHOW';
%*HOW<role> := 'RoleHOW';
my %*PKGCOMPILER;
%*PKGCOMPILER<role> := Perl6::Compiler::Role;
<sym> :my $*PKGDECL := 'grammar';
<package_def>
}
+token package_declarator:sym<diagram> {
+ <sym> :my $*PKGDECL := 'diagram';
+ <package_def>
+}
token package_declarator:sym<role> {
<sym> :my $*PKGDECL := 'role';
<package_def>
GrammarHOW is defined in src/metamodel/GrammarHOW.pir. I just took that file, replaced every occurrence of Grammar with Diagram, and saved the result as src/metamodel/DiagramHOW.pir. Since GrammarHOW.pir made reference to a Grammar class in src/core/Grammar.pm, I created a Diagram class and saved it in src/core/Diagram.pm.
In src/Perl6/Actions.pm I had to make a corresponding addition:
method package_declarator:sym<module>($/) { make $<package_def>.ast; }
method package_declarator:sym<class>($/) { make $<package_def>.ast; }
method package_declarator:sym<grammar>($/) { make $<package_def>.ast; }
+method package_declarator:sym<diagram>($/) { make $<package_def>.ast; }
method package_declarator:sym<role>($/) { make $<package_def>.ast; }
method package_declarator:sym<does>($/) {
Usually when I made a false step and got an internal error (not shown here), it was because I had neglected to add something to
src/Perl6/Actions.pm.
element declaratorAdding this to src/Perl6/Grammar.pm:
token term:sym<routine_declarator> { <routine_declarator> }
token term:sym<multi_declarator> { <?before 'multi'|'proto'|'only'> <multi_declarator> }
token term:sym<regex_declarator> { <regex_declarator> }
+token term:sym<element_declarator> { <element_declarator> }
token term:sym<circumfix> { <circumfix> }
token term:sym<statement_prefix> { <statement_prefix> }
token term:sym<**> { <sym> <.panic('HyperWhatever (**) not
| '(' ~ ')' <signature> <trait>*
| <routine_declarator>
| <regex_declarator>
+ | <element_declarator>
| <type_declarator>
]
}
+proto token element_declarator { <...> }
+token element_declarator:sym<element> {
+ <sym> {*} #= open
+ :my $*METHODTYPE := 'regex';
+ <element_def>
+}
+rule element_def {
+ [
+ { $*IN_DECL := '' }
+ <deflongname>?
+ <.newpad>
+ [ [ ':'?'(' <signature> ')'] | <trait> ]*
+ {*} #= open
+ '{'[ '<...>' |<p6regex=.LANG('Regex','nibbler')>]'}'<?ENDSTMT>
+ ] || <.panic: "Malformed element">
+}
And this to src/Perl6/Actions.pm:
if $<variable_declarator> { make $<variable_declarator>.ast }
elsif $<routine_declarator> { make $<routine_declarator>.ast }
elsif $<regex_declarator> { make $<regex_declarator>.ast }
+ elsif $<element_declarator> { make $<element_declarator>.ast }
elsif $<signature> {
my $list := PAST::Op.new( :pasttype('call'), :name('&infix:<,>') );
my $decls := $<signature>.ast.get_declarations;
+method element_declarator:sym<element>($/, $key?) {
+ if ($key) {
+ my %h;
+ %REGEX_MODIFIERS := %h;
+ } else {
+ make $<element_def>.ast;
+ }
+}
+method element_def($/, $key?) {
+ make PAST::Stmts.new();
+}
+class Perl6::DiagramActions {
+}
+
At this point, element declarations just called into the normal regex sublanguage. Needed to change this as a final step.
Perl6::Diagram sublanguageFinally, we make these changes to src/Perl6/Grammar.pm:
my %*LANG;
%*LANG<Regex> := Perl6::Regex;
%*LANG<Regex-actions> := Perl6::RegexActions;
+ %*LANG<Diagram> := Perl6::Diagram;
+ %*LANG<Diagram-actions> := Perl6::DiagramActions;
%*LANG<MAIN> := Perl6::Grammar;
%*LANG<MAIN-actions> := Perl6::Actions;
- '{'[ '<...>' |<p6regex=.LANG('Regex','nibbler')>]'}'<?ENDSTMT>
+ '{'[ '<...>' |<p6regex=.LANG('Diagram','expr')>]'}'<?ENDSTMT>
+grammar Perl6::Diagram is HLL::Grammar {
+ rule expr { <thing> ** 'on' }
+
+ token thing {
+ [
+ | <quote>
+ | <attribute>
+ | <call>
+ | 'vline'
+ | :s 'table(' <expr> [',' ':'\d*<identifier> ]* ')'
+ ]
+ }
+
+ token quote {
+ [
+ | <?[']> <quote_EXPR: ':q'>
+ | <?["]> <quote_EXPR: ':qq'>
+ ]
+ }
+
+ token call {
+ '<' <identifier>
+ [
+ | '>' :s [from <attribute>]?
+ | :s [ <identifier> '=' <quote>]* '>'
+ ]
+ }
+
+ token identifier { <.ident> [ <[\-']> <.ident> ]* }
+
+ token attribute { '.'<.identifier> }
+}
The above grammar parses my BookLayout example above. I guess the cool thing isn't that it does, but that it parses it inside a regular Perl 6 script. That's what this was all about, after all. The technology is there in Rakudo; it's just not user-accessible yet.
Doing this today is not easy. I stumbled upon maybe ten types of internal error along the way, about half of which I never managed to diagnose, but simply got rid of by doing things again, more slowly.
Summarizing the situation:
Yes, extending the syntax is possible. It's not particularly easy. That's just because the Perl 6 vision isn't fully implemented yet. But there is a roadmap, and we're sticking to it.
None of the internals hacking will be needed once this works the real way. I'll still have to modify the grammar and the actions, and I'll still have to add a DiagramHOW metaclass, but I'll do it from the safety of a perl script or a module, not by prying open the Rakudo source files.
I'm sure there's a whole heap of errors that could be removed or improved here. Problem is, my usual instinct for submitting bugs gets put on hold when I'm doing naughty things like internals hacking. What would I report it as? "[BUG] Nasty error message after applying this patch to Rakudo"? Somehow that doesn't feel right.
After getting diagrams to parse, I have no idea whatsoever how to give them the right semantics. What grammars do is to emit the regexes as specialized PAST nodes. There are no specialized PAST nodes for diagrams.
Looking forward to seeing this get much easier over time.
Last weekend I was at OSDC.fr. Amongst the nice food, free Hoegaarden (thanks GitHub!) and meeting people, I gave a couple of talks. One was the tried and tested Perl 6 Signatures talk, which was well received. The second was new, and mostly about 6model.
The slides from the 6model talk stand alone quite well in most places, and are perhaps the best overview you can get of 6model so far. However I want to take a moment to elaborate on a couple of them, because while they were good enough visual aids for what I was saying, they don’t quite tell the whole story. I’d also like to fill out some details I skipped over.
I talked a bit about the concept of representation polymorphism. I gave the example of a class representing color information:
class Color::RGB is repr(*) {
   has uint8 $.red;
   has uint8 $.green;
   has uint8 $.blue;
}
This is a case where your use case for the class may motivate alternative approaches to storing the data. If you’re going to store hundreds of thousands of these in an array of some kind, you’ll probably wish for a bit-packed representation in order to minimize memory consumption. On the other hand, you may be passing it outside of your program to a C library and want it to match the memory layout of a C struct so the cost of marshaling is low.
The ability to use a single class definition with different memory layouts is called representation polymorphism. It’s something that’s in the Perl 6 spec (see S12). It’s also something I chose to ignore in he earlier days of Rakudo – there were so many more important things to focus on in order to build an interesting subset of Perl 6 for folks to play with. Now we have that, though, and I’m seeking to design a model that will give us all we need to get to 6.0.
So how does 6model handle this? I took all of the various things you may want to do involving an object, and divided them up into two camps: those things that are tied to the representation, and those that aren’t. From there, I derived two APIs: a REPR API (which contains all the things that relate to in-memory representation) and a HOW API (all other aspects of the higher order workings of an object). The REPR API cares about:
Everything else is in the HOW API. That includes the obvious things, like declaration, method dispatch and introspection. It’s worth mentioning that add_attribute (the thing we call when we spot an attribute declaration in a package declaration) is in the HOW API, not the REPR API. This is because possession of an attribute is decoupled from storage strategy for it (getting the separation of concerns for attribute-y bits right is key to getting representation polymorphism right).
Therefore, each object has a couple of things that define its semantics: something that implements the HOW API and something that implements the REPR API. For various other reasons, we also want to independently reference the type object and keep a v-table pointer around outside of the meta-object (there’s a couple of design decisions that need their own blog posts here – bear with me).
We’d really like objects to be fairly lightweight, however. And since for most classes there will only ever be one REPR that they use, having pointers to both in every single object is rather wasteful – not to mention the extra stuff I mentioned wanting to have to hand. So, where do we stick it?
As the David Wheeler quote goes, “any problem in computer science can be solved with another level of indirectionâ€. So I added one: an object points to a “shared tableâ€, known from here-on-in as an S-Table. We have one S-Table per (HOW,REPR) pair that is in use. S-Tables are not objects, they’re just a little glue between others. (The less often quoted follow-up line to the Wheeler quote is, “but that usually will create another problemâ€. In this case, we’re just making a trade-off: a pointer dereference for saving memory – but read on for why that may not matter too much).
Armed with this, you can now understand the diagram on slide 82. I’d just like to add one clarifying note – the meta-object (implementing the HOW API – top right) is just a plain old object. It doesn’t have a special memory layout that is a set of method pointers; it too just references an S-Table and its memory layout is controlled by some REPR. I did the diagram that way because I wanted to emphasize the methods that object supported, and later realized that it could be confusing – sorry.
Finally, I’d like to throw out a couple of thoughts on the REPRs, just as bullet points rather than in full detail.
Sometimes things seem to click and fall into place inside my brain. Previously unrelated concepts turn out to be have commonalities after all.
For example, I said yesterday on #perl6 that I'd realized, teaching Perl 5,
that taking away wantarray was only a logical next step after taking away
sigil variance. Moritz was way ahead of me, though: he had blogged about
exactly
this
a year ago.
But a much deeper insight hit me on the Sunday, when jnthn++ was talking, and
explaining how Captures and Signatures form two sides of a function call
binding:
caller callee
Capture <---> binder <---> Signature
and then showing a slide that looked like this:
my ($some, $random, $things) := some-function();
The point of the slide is that Signatures aren't just used in function
declarations; they're used in the above case as well. I've always thought that
this was a random, fortuitous bonus use of it...
...but then Liz turned to me and whispered "that's because return is just
another function call". And right then and there, I went through some sort of
mini-enlightenment. How beautiful!
Later, Liz divulged that this was the kind of insight that came naturally when
one had programmed in continuation-passing style
for a while. It'd be interesting to learn whether CPS is the reason for
Signatures being used in bindings like this in Perl 6. For some reason, my
guess is no.
Anyway, Perl 6 surprising me with its consistency is only appropriate, given the name of this blog.
Things I will remember from the trip to Paris:
Sleeplessly hacking away at presentation software at night at Liz and Wendy's place. (This was after a fantastic pancake dinner and a set of competitive games of Martian Fluxx, most of which were won by Liz.) At about 2 o'clock I managed to get the "viewport" in the presentation to pan over from one slide to another, and then discovered that it all looked smoother when a sine function was applied to the "reaction coordinate". ((sin(πx - π/2)+1)/2 maps [0,1] to [0,1].) I the found that it looked even nicer when the sine function was applied twice.
Giving my "Hardcore bug reporting" talk and enjoying doing so. I believe I will give this talk again in some form or other.
Paris (or at least the part we were in) being unexpectedly derelict. No complaints about the venue, the hotel, the people, or the food, which were all excellent. Just the city itself being ugly beyond my wildest dreams.
jnthn and mberends sleeping very soundly after an active first day while I sat cross-legged on my hotel bed, debugging presentation software and franticly writing slides for my other talk. (Incidentally, I hear there's a way to actually finish making the slides before the day you're supposed to give the talk. It's called "using slides from a previous conference", and I don't do that nearly enough.)
Giving my other talk, "Perl 6: there's some madness to the method after all". I feared I might have too few slides, but instead I ran out of time and had to rush things a bit towards the end. This second talk didn't feel like a big splash, but might still have a few ideas in it that could be polished into something good. I kinda like the 'badges' idea.
French jumping back into my head, annoying me with the readiness by which my brain could parse it compared to its almost non-existent ability to produce it. As a small consolation, by the time we handed Liz back to Wendy, we seemed to have managed to corrupt the former into speaking English with a heavy lolspeak accent, almost to the same extent as jnthn, mberends and I.
A few hours ago colomon blogged a Perl 6 solution to the longest palindrome programming challenge. It was a bit slow.
An all-regex solution looks like this:
for $string.match(:g, /(\w+) \w? <{ $0.flip }> /) { # process palindrome match here # filter out the longest match }
It's also very slow, and it's very wasteful. So, can there be a hybrid between manual search a regex?
The regex engine is quite fast, and using it to find the center of a palindrome is a good starting point. From there on we can can move to both sides, and compare character by character if moving away from the match still results in a palindrome:
my $s = 'Fourscoreandsevenyearsagoourfaathers...'; my $solution; my $longest = 0; for $s.match(:g, /(\w)\w?$0/) { # $_ now holds a Match object, my $left = .from - 1; my $right = .to; # go to left and right while palindromic while $s.substr($left, 1) eq $s.substr($right, 1) { $left--; $right++; } # we move a bit too far $left++; $right; # and we're only interested in the longest # palindromic substring my $len = $right - $left; if $len > $longest { $solution = $s.substr($left, $right - $left); $longest = $len; } } say $solution;
This now runs in under 1.5 seconds on the 1169 characters long example input string.
So, the third challenge was to count the number of subsets of a set of numbers such that the largest number in the subset is the sum of the rest of the numbers in the subset. My first attempt was very straightforward: create all the subsets and check to see if they have the desired property:
my @a = 3, 4, 9, 14, 15, 19, 28, 37, 47, 50, 54, 56, 59, 61, 70, 73, 78, 81, 92,
95, 97, 99;
my $hits = 0;
for 1..(2 ** +@a) -> $key {
my @b = gather for 0..+@a -> $i { take @a[$i] if $key +& (2 ** $i); }
my $test = @b.pop;
next if $test != [+] @b;
$hits++;
say (@b, $test).join(' ');
}
say "$hits hits";
I think this works correctly, but it will be a long time before we know — as I type this, it’s been running for ten hours on my fastest computer, and I don’t anticipate it finishing any time soon.
My second attempt relies on recursion, the fact the list is sorted, and skipping fruitless branches to get the job done much faster — 47 seconds, to be precise.
sub CountSumSets($target, @a) {
my $sets = 0;
for ^ +@a -> $i {
if @a[$i] < $target {
$sets += CountSumSets($target - @a[$i], @a[($i + 1) .. (@a - 1)]);
} elsif @a[$i] == $target {
$sets += 1;
}
}
$sets;
}
my @a = 3, 4, 9, 14, 15, 19, 28, 37, 47, 50, 54, 56, 59, 61, 70, 73, 78, 81, 92, 95, 97, 99;
@a .= reverse;
my $hits = 0;
for ^ +@a -> $i {
$hits += CountSumSets(@a[$i], @a[($i + 1) .. (@a - 1)]);
}
say $hits;
Via Hacker News, I found the Greplin Programming Challenge, and couldn’t resist trying it in Perl 6. If you’re the sort of person who enjoys that sort of thing, I highly encourage you to stop reading here and go try it yourself!
I’m going to blog about all three challenges one-by-one. I suppose the third will be the most interesting, if for no other reason than my current implementation looks like it will take ~32 hours to run, so I’m probably going to need to find a more clever solution.
Basically, the first challenge is to find the longest palindrome in a string without spaces. As stated, I thought the challenge implied it was case-sensitive; obviously it’s easiest enough to add a call to lc to get a case-insensitive version.
I was pretty sure a naive version would bring Rakudo to its knees, so I tried to be slightly clever. My solution is still O(N^2), but N is the number of occurrences of a given letter rather than the full length of the string, so that’s fairly reasonable.
sub longest-palindrome($string) {
my @c = $string.comb(/./);
my %hc;
for ^ +@c -> $i {
if %hc{@c[$i]} {
%hc{@c[$i]}.push($i);
} else {
%hc{@c[$i]} = [$i];
}
}
my @candidates := gather for %hc.keys.sort -> $c {
say "Checking $c";
my $list = %hc{$c};
say :$list.perl;
for $list.list -> $i1 {
for $list.reverse -> $i2 {
last if $i2 <= $i1;
my $j1 = $i1;
my $j2 = $i2;
my $candidate = Bool::True;
while ++$j1 < --$j2 {
if @c[$j1] ne @c[$j2] {
$candidate = Bool::False;
last;
}
}
if $candidate {
say @c[$i1..$i2];
take @c[$i1..$i2].join('');
}
}
}
};
@candidates.sort({$^a.chars <=> $^b.chars}).perl.say;
}
Basically, my notion was to store all the occurrences of each letter in a hash of arrays, then pair up every two occurrences of the same letter and see if they are a palindrome. This probably isn’t the most elegant solution, nor the fastest, but it was fast enough to get solve the challenge problem in a minute or two, and easy enough to code I could do it in under an hour at three in the morning.
Interestingly, I think there might be a way to solve this using regular expressions…
Update: Moritz has a solution which blows this one out of the water, both much faster and more elegant. (The key idea was using the regex engine to find the center of potential palindromes.) I’ll let him tell you about it…
For this week's contribution to Perl 6 we ask you to implement the
Str.samespace method for Rakudo. This method takes a whitespace
pattern from one string, and transfers it to a second string. It is used for
smart substitutions.
(Introduction to this series of challenges)
The Perl 6 specification includes a variant of the substitution operator
that preserves the spaces from the source string. This is activated either the
:ss/:samespace adverb, or by using
ss/// instead of s///.
Implement a samespace method that can act as the backend for
such an operation.
Since recompiling Rakudo takes a lot of time, you'll have a much easier time developing the method in a separate file, and simply run it with Rakudo. You can use this template for prototyping:
use v6; use Test; use MONKEY_TYPING; augment class Str { method samespace($other as Str) { # replace this by your implementation uc self; } } plan *; is "a b c d".samespace("x y\n\tz"), "a b\n\tc d";
The Specification has
a few more details on the topic. (Ignore the part about the whitespace
having to be matched by a <ws> rule). Your implementation
doesn't have to fulfill the spec totally; a naive implementation of a small
part is still much better than none at all.
Please also add a few more tests.
For extra karma you can make your submission a real patch (in which case
the method goes into src/core/Cool-str.pm). More extra karma and
eternal fame if
you wire it up with the :ss and :samespace adverbs
in the subst method (same file).
Please submit your source code to the [email protected] mailing list (and put [email protected] on CC, because the mailing list sometimes lack quite a bit).
Update: There have been two submissions so far, challenge closed.
If you have any questions, feel free to send them to the mailing list, or ask on our IRC channel".
My projects are rotting. Fast.
The really big change was the transition to a new Rakudo (known as ng) in January, of course. I was one of the people who stuck to the old Rakudo (known, retroactively, as alpha) for the longest time, because
alpha simply won in terms of features, and it wasn't feasible to apply workarounds to make code work in ng;alpha and not having the carpet move under me for a while;ng started to significantly overtake alpha, it was the hugeness of the task that daunted.So far, I've ported over GGE (grammar engine) and Yapsi (Perl 6 implementation). Projects that remain to be ported over include November (wiki engine), Web.pm (web framework), perl6-literate (runnable blog posts), pun (-n and -p flags for Rakudo), and Tardis (time-traveling debugger).
I ported GGE from alpha to ng on and off over a long period of time. Without the test coverage I have for that project, I probably would have given up. Just to give you an idea of the amount of work it was, the changes to bring GGE from alpha to ng can be found here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, and here. For those who want just the good bits already picked out, I once wrote a blog post on the lessons of porting GGE.
I ported Yapsi from alpha to ng mainly by rewriting the whole of Yapsi over the course of a few days. The change went in as a single commit declaring that "Yapsi now has a new Yapsi". Fitting, since Rakudo now had a new Rakudo.
I've started porting Druid from alpha to ng. I stopped because I hit some trouble which I had to work around, then hit some more trouble which I had to work around, at which point I hit some serious trouble. And it wasn't that ng had bugs or did things wrong either, it was that Druid used aspects of alpha (mainly in the grammar engine) that simply didn't exist anymore. I now think that Druid needs a mid-sized redesign; the old design worked under alpha, but for ng Druid needs something new.
It's not just the alpha-to-ng transition. Last night while going by car to visit Amsterdam.pm, I picked up a two months old slides-generating script, thinking I might use it to whip up a presentation for the evening. Turned out it had a regression bug and another regression bug. This is exactly the effect the Jargon File describes as bit rot:
Instead of trying to work around the bugs in the car, I stashed two bug reports and eventually made a talk around the script instead. (And I didn't have to run it.)
Bitrot happens quickly in Rakudo applications. This basically tells me three things:
It could perhaps be seen as a tad ironic that in wanting to help create a tool chain for Perl 6, I have instead fallen victim to the lack of it. Oh well; live and learn.
Once I was reasonably happy with ABC::Duration and ABC::Note, I went ahead and added ABC::Rest, which was truly easy thanks to the magic of code reuse. It actually turned out to be quite a struggle to implement, but that was only because I accidentally typed is $.type rather than has $.type and spent a couple hours trying to sort out what the LTA error message meant.
class ABC::Rest does ABC::Duration {
has $.type;
method new($type, ABC::Duration $duration) {
self.bless(*, :$type, :ticks($duration.ticks));
}
method Str() {
$.type ~ self.duration-to-str;
}
}
There are several sorts of rests in ABC, which is why ABC::Rest has a $type attribute. In practice, I’ve only implemented “z†rests so far, as they are the basic type.
Rests were already in the grammar, and adding an action to support them was dead easy:
method rest($/) {
make ABC::Rest.new(~$<rest_type>, $<note_length>.ast);
}
After a trivial refactor to make the Lilypond duration handling code easily available, it was just took one line to add rests to the Lilypond output:
[/sourcecode]
when "rest" { print " r{ DurationToLilypond($context, $element.value) } " }
[sourcecode]
That’s for the output section of the code. No changes were required to make rests work in the “figure out the overall duration†section of the code, because it looks for elements which do ABC::Duration, and so it automatically found the rests and correctly handled them.
To me, this looks like a solid win for treating duration as a role.
So, I just dived in and started trying to implement duration as a role.
role ABC::Duration {
has $.ticks;
our multi sub duration-from-parse($top) is export {
ABC::Duration.new(:ticks($top.Int || 1));
}
our multi sub duration-from-parse($top, $bottom) is export {
if +($top // 0) == 0 && +($bottom // 0) == 0 {
ABC::Duration.new(:ticks(1/2));
} else {
ABC::Duration.new(:ticks(($top.Int || 1) / ($bottom.Int || 1)));
}
}
our method Str() {
given $.ticks {
when 1 { "---"; } # for debugging, should be ""
when 1/2 { "/"; }
when Int { .Str; }
when Rat { .perl; }
die "Duration must be Int or Rat, but it's { .WHAT }";
}
}
}
The corresponding action for an explicit note length is
method note_length($/) {
if $<note_length_denominator> {
make duration-from-parse($<top> ?? $<top>[0] !! "", $<note_length_denominator>[0]<bottom>[0]);
} else {
make duration-from-parse($<top> ?? $<top>[0] !! "");
}
}
I messed around a bunch trying to make "e" does Duration work as a type, but eventually gave up and just coded an ABC::Note type:
class ABC::Note does ABC::Duration {
has $.pitch;
has $.is-tie;
method new($pitch, ABC::Duration $duration, $is-tie) {
say :$duration.perl;
self.bless(*, :$pitch, :ticks($duration.ticks), :$is-tie);
}
}
with corresponding action
method mnote($/) {
make ABC::Note.new(~$<pitch>,
$<note_length> ?? $<note_length>[0].ast !! ABC::Duration.new(1),
$<tie> eq '-');
}
So… that seems to work okay so far. But it does raise some issues for me.
1. I’m finding this $<top> ?? $<top>[0] !! "" pattern to be very repetitive. Surely there must be a better way to do it? (errr… wait a minute, could that be just $<top>[0] // ""?)
2. I don’t mind the proliferation of small classes in my code, that corresponds nicely to what we are modeling. But I am starting to mind the corresponding proliferation of small source files. Is there a better way to organize things?
… and that’s all I can remember at the moment. I think I’ll wander off and have breakfast.
So, I patched in some dodgy but working for this example code to handle the duration of notes. And then I started to think about how to do it properly.
About half of the musical elements in an ABC tune have a duration. So my first thought was, oooo, define an ABC::Duration role! I was excited, because this seemed like the first really good example I’d seen in my own work of where a role was appropriate and seemed superior to class inheritance.
So I started out defining a duration role with a Real attribute for its length, and two new methods, one which took a Real, and one which took the parsed bits of the note_length regex as strings. But then I started thinking about what that meant. If you define a class which does a role, how do you call that role’s new method? If I’m building up the AST, I’m going to want to create a duration before I have an element to attach it to — how do you build an element around an existing duration? And half of the objects which have durations are perhaps not best represented using a stored duration value — for instance, a triplet element’s duration is the total duration of its three wrapped notes, times 2/3 — making the stored duration value redundant.
So, this is one of those posts which I make before I’ve figured out what the answer is. I’ve verified that you can create a standalone role object in Rakudo:
> role Duration { has $.duration; }; my $a = Duration.new(:duration(10)); say :$a.perl
"a" => .new(duration => 10)
So that’s one piece of the second point addressed. (Though surely that .perl is wrong?) But I’m not seeing how to do the rest of it at the moment.
Any ideas out there?
I'm starting to see a pattern in more and more places of programming. I haven't looked too hard for its real name — and it probably has one, as common as it is — but for now, I can think of several: the compiler pattern, or generate, transform, serialize, or it's just a tree, silly.
If I am first in formulating this as a pattern, then... wohoo! But I strongly doubt that I am.
Here's how it works: at the start of the program, something reads some input and converts it to an internal format called, for the purposes of this post, the internal format. Then zero or more dedicated parts of the program massage the data, which is now in the internal format. Finally, something converts the whole thing to a serialized form an outputs it. That's it: input, massage, output. Or, if you wish, generate, transform, serialize.
The pattern should be familiar to Perl programmers. Perl is, after all, an extraction and reporting language, and professes to perform at least these two tasks in a practical way. In the case of Perl, the internal representation consists of data structures in the form of arrays and hashes.
In a previous life, I helped manage a web app written on top of Apache Cocoon, through which I learned the triplet generate, transform, serialize. In the case of Cocoon, the internal representation is XML. Cocoon is one of the few programs out there that actually uses XML for something significantly more than hype and cargo culting. It's worth checking out. The generators, transformers and serializers in Cocoon are the actual primitives, and changes to the web app are made in a sitemap file containing the way these primitives are piped together.
A compiler can be seen as these three steps. There's even a ready-made vocabulary for them: front end, middle end, and back end. The internal structure is a tree representing the nested regions of the program text, or (later) possibly a more abstract tree representing actions and control flow. One example is Yapsi, our p6-implementation-in-p6. There's a grammar at the front end parsing the text into a tree of Matches. Little top-downers and bottom-uppers climb around this tree and decorate it with important information. Then the tree is flattened out into flat text-y SIC and shuttled off to the runtime.
Or take Hitomi, the not-quite-finished template engine that lets you mix XML and Perl code, leveraging the power of both. The secret recipe it uses to make this possible... is the patterns again: read in the original template at one end, parse it up into an XML stream, let the stream flow joyously through various filters before finally being serialized as text at the other end.
Even a small utility program that I wrote years ago to convert a bunch of money transactions to a final tally of who owed whom what qualifies for this pattern, though only barely. See, it read in the format (in YAML), represented it internally as a directed weighted graph, did a number of simplifications on the graph until it could get no simpler, and finally spat out the result (as YAML).
Why is this pattern so common? This seems to be a difficult question to answer until one realizes that the alternative, not to convert to the internal format, is usually much, much worse. In all of the above cases, the data is lifted from its original format, it's freed from the shackles of text, and clad in beautiful clothes made of the purest trees, graphs, streams and other structures that allow the programmer not only to see the things she is manipulating more clearly, but also to think more clearly about the programmatical units doing the manipulating. In freeing the data from the text, she frees herself from death by a million substr calls.
In some sense, Postel's law is showing through here as well.
By having to pass through the fire test at the beginning, we make sure that nothing of the liberalness remains of the input that we accept into the sensitive inner workings of the program (perhaps because they can't even be represented by the internal format), and knowing that what we have after that point is a very exact version of the data allows us to be very flexible in manipulating it. The strictness in the output follows as a natural consequence of the internal format being strict. Cocoon will never output mismatched XML tags, unless perhaps you write a faulty serializer.
It was the following last example that made me think of all this: in order to rescue the blog content from use Perl to my new blog, I decided to download it all and massage it with a Perl 5 script. Source format: use Perl's idiosyncratic HTML soup. Target format: Markdown. Three, two, one, go!
What I ended up writing was a while (m//g) { ... } loop that found start and end tags for me, thus tokenizing the input into tags and their intervening text content. Friendly hash references made sure things ended up in the right places in a tree. Presto, an explicit tree made from the implied tree of the HTML tags.
(It was a this point that I started noticing that use Perl produces opened-but-never-closed <div> entities each time someone uses the made-up <quote> entity in a blog post. Easy enough to work around, but... odd that no-one has noticed, during all those years of Slashcode.)
Next step: serialize the tree down into Markdown. This step contained a lot of heuristics, some catering to Markdown's preferred number of escapings in various contexts, some adding post-facto richness that never was, and never could be, in the original use Perl-crippled HTML. It was fun work, because what this script was a lot of blog posts that I had written in insane HTML, now converted to sane Markdown.
It was only after I was done that I stopped to wonder why I had chosen to go the way of a tree when doing this. Couldn't I have skipped the middleman and just converted HTML tags to Markdown directly?
And here's the thing: no, I don't think so. I don't think I would have finished the task had I chosen to solve it without the intermediate tree structure. Because it would have come down to parsing a lot of text with increasingly crazy regexes, some of which would have had to take into account deep nesting... instead of handling the deep nesting using subroutines recursing over a tree.
So, yes, it's just a tree, silly. Or a directed graph, or a DOM, or a stream. But often that's just what's needed to make the problem manageable.
For a long time, I kept using use Perl for my blogging, knowing that there are better solutions out there. blogs.perl.org being among the most alluring alternatives. Various Perl 6 bloggers were also reporting happiness with Wordpress-based blogs. Inertia kept me tied to use Perl.
Inertia — and something else. I had a teeny tiny dream that I could make a blogging solution in Perl 6. That would, I theorized, make me ultimately happy.
Except that it would never happen. It's quite a big step to move blogs, and doing it while actually inventing the new blog under oneself... no wai.
And then use Perl forced my hand. After ~10 years of service, it has now been shut down. And suddenly, I found myself without a blogging platform.
So I thought, what the heck.
A couple of weeks later, here I stand with a working blogging solution in Perl 6. Yay!
I'll have a lot more to say about the blogging software as we go along, but here are the main points in bullet form:
In summary: yay, new blog! Watch this space.
It is my pleasure to announce try.rakudo.org, an interactive Perl 6 shell in your browser, based on the Rakudo Perl 6 compiler.
It gives you the opportunity to try out small pieces of Perl 6 code without having to install a compiler yourself, and show the results to your colleagues.
Future plans for try.rakudo.org include a tutorial, which gives you instructions that you can try out immediately.
Credits for creating the site go to John "ash" Harrison and cygx; hosting is provided by Juerd Waalboer on feather.
Update: after publishing this post I've learned that the main author already wrote an announcement, which is also worth reading.
I know that, as well as wanting a faster, more complete Rakudo, many people watching my work on Perl 6 are keen to understand and learn about the areas that I’m currently hacking on. Therefore, during my current grant, I’m going to write a various blog posts to try and explain what a lot of the terminology I throw around really means. This is the first of them, and I’m going to start by talking about three topics which are very closely connected to each other, even if they may not seem so at first. Namely, they are:
Perl 6 is neither statically nor dynamically typed, rather it is gradually typed. For example, consider the following program:
sub get_cat_from_rescue_center($type, $owner) {
   my Cat $rescued = cat_search($type);
   $rescued.owner = $owner;
   return $rescued;
}
my $kitteh = get_cat_from_rescue_center('tabby', 'Anna');
Putting aside any type inference that we may be able to do in the future, what do we actually know about the types of the variables in this program?
So is this information of any use if we want to make our programs run faster? The answer is a resounding “yesâ€. For any given concrete type, which may have some parents, we can – at compile time – compute a v-table. A v-table is simply an array of code objects. Along with that, we make a mapping of method names to slots. Now, when we compile a method call on an object, if we know its type and its slot mapping, we can look up the method in this mapping and emit a call based on an array index lookup.
“Well,†you may be thinking, “that’s all very well for the programs where I declare types, but I never do that, so I win nothing, right?†Wrong. Take a look at the methods in Mu and Any, for example. Methods like new, ACCEPTS, Str, grep, map, list – in other words, methods that are either called a lot for you when you use various Perl 6 features, or that you will often use yourself. You will get a win every time you call .new to create a new instance of an object, every time you do a smart-match, etc.
Of course, you can get further wins by providing more type information. And, of course, the various internal bits of Perl 6 – which we implement in Perl 6 and NQP – can benefit from gradual typing too. In general, the rule is “the more type information you provide, the more information you give to the compiler in order to generate better code and maybe catch errorsâ€.
This is, of the three things I mentioned at the start of this post, the “goalâ€. So now let’s look at how to make it happen.
What is a class? What is a role? What is a grammar? In Perl 6, these things don’t just magically have the semantics that we associate with them. Instead, they all have associated meta-types that define what they mean. These tend to be named, by convention, with “HOWâ€, short for “Higher Order Workings†but also nicely mnemonic: “Just HOW the heck does this type of thingy work?†When a class, or a role, or a grammar is created, we grab the appropriate meta-object and then call a sequence of methods on it to define the package. Let’s look at a quick example.
class Band {
   has $!name;
   has $!genre;
   method describe() {
       say "$!name play $!genre music";
   }
}
When we see a declaration like this, it really boils down to to something along the lines of:
::Band := ClassHOW.new_type();
Band.^add_attribute(Attribute.new(name => '$!name'));
Band.^add_attribute(Attribute.new(name => '$!genre'));
Band.^add_method('describe', method () { ... });
Band.^compose();
Here, ClassHOW is the meta-object that describes how default Perl 6 classes work. The set of methods that we can call on this object – or an instance of it, since ClassHOW is really a type object – is called a meta-object protocol. Put another way, the meta-object protocol (from now on, I’m just going to say MOP) is just the API that our meta-objects all implement. You add a method to a role, or a grammar, or a class, or whatever else, by calling an add_method method and passing along a name and an object representing the code.
The interesting question is when we do all of these method calls. Today in Rakudo we:
Essentially, this means we have two MOPs – one used at compile time and one used at runtime. Representing types differently at runtime and compile time is not unusual in compilers, not least because many do type erasure as part of the compilation process. It’s served us reasonably well so far in Rakudo too. However, it has some disadvantages. Of note:
Thus for both performance reasons (duplicated work is bad) and the fact that we’re at the point where we do want to implement optimizations and more compile-time checks (which means more duplicated information), I think it’s important that we drop having separate MOPs at compile time and runtime, and instead have one unified MOP.
This means that when we parse the opening curly of your class declaration, we really do at that point create the very same type object and meta-object that will exist at runtime. We really do a trait dispatch to trait_mod:<is> at compile time to add the parent class – the type-object for which we’ll already have to hand. We really do call add_method on the meta-object after parsing and making the AST for a method (though it’ll be passed a code object that still needs to be filled in with the compiled code later on – we can’t actually invoke the method until runtime). And at the closing curly of the type, we really do call the compose method on the meta-object (conjectural: we may actually need to defer this until CHECK time, when we’ve parsed and made AST for the whole compilation unit; the point is that it’s somewhere at compile time).
So if it’s so wonderful, why wasn’t it done like this in the first place? Of the various reasons, the biggest is that it requires a significant change to the overall compilation architecture, and the implementation of something that we’re completely missing today. Put another way, we couldn’t have done this before without it severely impacting other goals that, at the time, were more important in getting Rakudo to be useful and usable. So what’s the secret sauce?
If objects are going to survive between compile time and runtime, in a situation where we can actually compile code and save the stored code somewhere, we’re going to need to be able to serialize them (or, in other terms, freeze them) and then later deserialize them (also known as thawing).
The difficult problem here is not so much serializing an individual object. It’s that we need to bound that serialization to things in the current compilation unit, and be able to reference objects that have come from other compilation units. For example, a meta-object for class Lolcat may need to refer to the type-object of its parent class Cat and a role Lol, which may both be from different (perhaps pre-compiled) modules.
Developing this kind of thing is non-trivial, and it doesn’t exist today in Parrot. However, I think at this point, having this kind of support is essential to the future of Rakudo. Thus, it’s going to be something I’ll dig into seriously in a week or two. There’s a lot of questions about how to achieve it, and various decisions I’ll have to take.
One notable question is whether I try and upgrade Parrot’s freeze/thaw support to support this, or if I just build something from scratch, as I am with the rest of the object model implementation. There are advantages either way, and of course disadvantages.
If I try to build on the existing Parrot implementation of freeze/thaw, I make this functionality trivially available to other language developers on Parrot. I also have part of what’s needed already in place, however the bounding and linkage bit (e.g. the hard part) is missing. Of note, it works for Parrot’s hash and array PMCs, which I am using, so I guess I save a little work there, but I’d still need to implement the serialization of the Perl 6 objects. A downside is that I’m not too optimistic about the freeze/thaw subsystem being in great shape, so I may have my work cut out for me to extend it (e.g. it will possibly need some refactoring first). I will have to change the bytecode format to support this, I will have to make various other changes, and I may well run into the Parrot deprecation policy. The deprecation policy is a well-intentioned attempt to create stability for language developers, but the upshot is that it sometimes prevents needed changes and improvements. The situation if that did happen for this work could be icky.
If I build something from scratch, then I have the freedom to make it look how I want, and don’t have to worry about refactoring an existing code base nor the deprecation policy. I can make it look similar to an implementation of the same thing on other backends, which would mean I could prototype it first in 6model if I wanted to and then easily port it, just as I have been doing with the rest of the object model implementation. However, it would not fit in so neatly with the rest of Parrot, particularly objects from other languages that made it into the object graph and needed serializing. That is, unless they decided to build on top of the same object support in nqp-rx that Rakudo will – not an unrealistic option, since I expect the meta-model implementation I’m working on will make it easier to implement object systems as other languages want them than the current Parrot OO support does. Not to mention that the same model will become available on other backends in the future.
At the moment I could go either way on this; I suspect the first step will be taking a deeper look at what the current state of Parrot’s freeze/thaw subsystem is, and perhaps doing a first cut of it in 6model (because it’ll be informative whichever way I go).
Anyways, I hope this post has been informative, and helped you get a better grasp of the three areas I’ve covered. I’ll be writing on more topics – including expanding on some things I’ve referred to in this post – but feel free to suggest topics in my grant work that you’d particularly like me to elaborate on by leaving a comment. In the meantime, it’s on with the hacking (and there’ll be a status report at some milestone point in the next week or two, all being well).
There were a number of comments on #perl6 on how to improve the code from my last post, but I’m ignoring them all today and forging ahead to get an actual ABC to Lilypond translator up and running. Here’s the core of the code:
sub StemToLilypond(Context $context, $stem) {
my $match = ABC::Grammar.parse($stem, :rule<mnote>); # bad in at least two ways....
my $pitch = ~$match<pitch>;
my $length = ~$match<note_length>;
print " { %note-map{$pitch} }{ %cheat-length-map{$length} } ";
}
sub BodyToLilypond(Context $context, @elements) {
say "\{";
for @elements -> $element {
given $element.key {
when "stem" { StemToLilypond($context, $element.value); }
when "barline" { say " |"; }
}
}
say "\}";
}
This code is wrong for a host of reasons. But the cool thing is, for simple tunes, it very nearly works. Basically, we go through the list of musical elements in the tune. When we see a barline, we output a bar line to Lilypond. When we see a stem, we parse it again (ugh) assuming it is the simple form a a stem, then map the pitch portion to the equivalent Lilypond note (ignoring key signature and accidentals), and the rhythm portion to the equivalent Lilypond duration (assuming that the ABC is notated in 8ths, and ignoring almost all potential complexity). This crude approximation almost suffices to properly notate “The Star of Rakudoâ€!
Here’s the PDF Lilypond outputs when feed the output of this script. At first glance this looks pretty good, but actually it’s got one major issue (as well as a host of minor ones). I think it might be obvious even to those who don’t read sheet music if it is compared to this proper PDF: the notes are all correct, but they are shifted one eighth note off in the measures! This is because Lilypond apparently treats bar lines in the source code as a mere debugging aid — if they don’t agree with what it thinks they should be, it will issue a warning and then go with what it thinks rather than what you said. I guess that behavior might make sense in big orchestral scores, but it is just annoying at this level.
So, what needs to be done to make this tune work:
1) Detect pick up bars and send the \partial command to Lilypond so that it keeps the bar lines in the correct places.
2) Preprocess the music looking for repeats, so that the repeated section can be properly marked for Lilypond.
3) Handle the key signature correctly.
4) Handle the ~ gracing, which in should generate a turn symbol over the note. (Used here to indicate an Irish-style roll.)
It appears to me that these are straightforward programming problems, so I’m going to forge ahead and see what I can do…
I liked the array of pairs structure for the header so much I decided to do the same sort of thing for the music part itself. Here’s the bit of the grammar this post talks about:
regex element { <broken_rhythm> | <stem> | <rest> | <gracing> | <grace_notes>
| <nth_repeat> | <end_nth_repeat> | <spacing> }
regex barline { ':|:' | '|:' | '|' | ':|' | '::' }
regex bar { <element>+ <barline>? }
regex line_of_music { <barline>? <bar>+ }
regex music { [<line_of_music> \s*\v?]+ }
As a crude first approximation, I transformed each element into a Pair with the name of the rule that made the element as the key, and the actual string parsed as the value. Likewise each barline becomes a Pair with the key “barline†and the string for the value. In the long run, I’m going to have an ABC::Element class (or something like one), but this version ought to be enough to get simple ABC translation up and running.
That all worked fine right from the start. But the next bit was trouble:
method bar($/) {
my @bar = @( $<element> )>>.ast;
@bar.push($<barline>>>.ast);
make @bar;
}
method line_of_music($/) {
my @line = $<barline>>>.ast;
my @bars = @( $<bar> )>>.ast;
for @bars -> $bar {
for $bar.list {
@line.push($_);
}
}
@line.push("endline" => "");
make @line;
}
method music($/) {
my @music;
for @( $<line_of_music> )>>.ast -> $line {
for $line.list {
@music.push($_);
}
}
make @music;
}
That’s what I ended up with, and it works. But when I started, each of those action methods was a simple one-liner. Alas, that version built up a structure in the results, when I wanted to end up with a nice, flat list of elements. I spent an hour trying different variations to remove that structure, finally ending up with the inelegant mess here. I’m hoping someone out there has a better way to do it.
At this point, this project is really close to being able to do useful things. Given a sufficiently simple tune (and many in ABC are very simple indeed!) this framework should make it easy to translate to Lilypond. I don’t know if I’ll get the chance to code it today, but if not, tomorrow for sure, I think….
I renamed the old ABC class ABC::Grammar. Then I created a simple ABC::Header class to represent the header of an ABC file. At its heart, an ABC::Header object is just an Array of Pairs, each of which consists of a header field name (like “T†for title) and a header field value. (Why an Array instead of a Hash? Because certain header fields can appear more than once, and order is important.)
Then I constructed a very simple actions set to test my understanding. For the two rules
regex header_field { ^^ <header_field_name> ':' \s* <header_field_data> $$ }
regex header { [<header_field> \v]+ }
I constructed the following actions:
class ABC::Actions {
method header_field($/) {
make ~$<header_field_name> => ~$<header_field_data>;
}
method header($/) {
my $header = ABC::Header.new;
for @( $<header_field> ) -> $field {
$header.add-line($field.ast.key, $field.ast.value);
}
make $header;
}
}
So when the parser constructs a header_field, it makes a Pair with the header field’s name as the key and its data as the value. The Pair is stored in the .ast for the header_field. Then when the parser makes a header, it constructs an ABC::Header object and fills it with the Pairs from the various header fields. Whee!
This is passing my initial tests, so I’m going to move on to some more complicated actions next…
I’ve been itching to get back to work on the ABC module (originally by SF from lastofthecarelessmen.blogspot.com, but my fork is the only version seeing current development work). A reference in an HN comment to Lilypond has provided the perfect. Lilypond is a long-lived open source project for music notation with very ambitious goals. In the past I’d tinkered with it a bit, but always come away unsatisfied. Today, though, I had a notion for an interesting project, so I thought I’d give it a whirl.
So, here’s a PDF of my tune “The Star of Rakudo†done with my usual slightly hacked version of jcabs2ps. And here’s a quick stab at a Lilypond version (with completely unnecessary 1st and 2nd endings added just so I can see what they look like). The Lilypond version has a heaviness to it I’m not wild about, but I think it’s pretty arguable that the various elements each look better than the other version. In addition, Lilypond appears to have good support for a more complicated musical elements. My inability to get jcabs2ps to portably generate all the musical elements I need has been a longstanding stumbling block to a side project of mine, so Lilypond is intriguing.
I don’t particularly like Lilypond’s music “languageâ€, but that’s okay. My idea is to have the Perl 6 ABC code automatically translate ABCs to Lilypond for me, in effect generating good-looking notation from them. It will then be the best of both worlds — the lightweight, easy-to-type ABC format generating professional-looking output from Lilypond.
So, now I need to figure out how add actions to a Perl 6 grammar…
So, I had a lovely vacation in Central Europe…
Lovely Ljubljana
…and now it’s back to the hacking! :-)
While the eventual goal of my current grant is, obviously, to deliver significant improvements to Rakudo, there are quite a lot of preliminaries that need to be taken care of first. My first steps have been to spawn a small research project in which I’ve done from-scratch prototype of how the Perl 6 meta-model could look, plus a few things around it to be able to actually run some code. The project, named 6model, currently consists of:
So, that’s where things stand today. So where are we going from here? I’d like to break this down into two parallel tracks, both of which I’m going to be working a lot on in the coming weeks.
The 6model project has been highly valuable so far for exploring design, and there’s a few things I want to explore further in it. My main research tasks here over the coming weeks will be (this is a rough ordering):
Now that I’m increasingly happy with various parts of the model I’ve explored and evolved in 6model, it’s time to start bringing it to the Parrot implementation of nqp-rx, as a pre-requisite to bringing it to Rakudo. However, there are a couple of other things that I expect to do first in Parrot nqp-rx, simply because they are dependent on having a bootstrapped NQP to do them.
Here is a rough outline of how I expect things to go, though some of these may well be re-ordered and they’re also in places parallelizable. I’ve broken it down into phases. Note that if your eyes glaze over at some of the terminology, don’t worry – I’ll blog a lot more on the details of many of these in the future.
Phase 1: get classes and grammars using the new meta-model
Phase 2: building meta-objects at compile time and supporting gradual typing
When we get to this point, we probably do have enough to start on getting Rakudo to use the new meta-model. However, it’s not the end of the changes that I think we’ll really want; next comes:
Looking a little further down the road (quite probably beyond getting my current grant done unless this suddenly becomes important), I also somewhat expect that nqp-rx may switch away from using Parrot’s String/Integer/Float/ResizablePMCArray PMCs to using real objects implemented in a low-level setting, using representation polymorphism, as shown in the example I linked earlier. That would further entail switch all operators over to be Perl 6 multi-dispatch subs. Rakudo has already done the latter and will certainly do the former in the new meta-model implementation. But it could well make a lot of sense – especially from a semantics and portability angle but maybe also a performance and usability one – to do this in NQP too.
So when will the nqp-rx branch that all this will take place in get merged (or possibly just become master)? Well, the answer is “probably not until Rakudo itself has evolved far enough to work on itâ€. This is going to entail a Rakudo branch, targeting an nqp-rx branch, for a while until we’re ready to switch them both.
This is, really, the same kind of depth of change as when we went from the first NQP implementation to nqp-rx. I’m not saying that we’ll end up with the mass regressions seen in Rakudo that were a necessary part of that process – I expect to pull this off with very little of that, simply because the nature of the changes here are very different. But a lot of things are going to look quite different after this process, and it’s not going to be without some pain. It is, however, a critical part of the path on the way towards Rakudo being a high quality Perl 6 implementation, and NQP and the compiler tools surrounding it being a high quality tool chain for implementing a language atop of a VM.
In C++, you can declare an attribute or a method as "protected". Which means that it is private, but subclasses can still access them.
This makes just as much sense as saying you only get access to these attributes if you wear a funny yellow hat.
I'll try to explain why.
Encapsulation has the advantage that you can change the implementation of a class without changing the API. If you write a library, that means you don't break your user's code if you change some internals.
However your users can not only instantiate your classes, but they can also inherit from them. Which means they can put a funny yellow hat on. There is no restriction as to who can inherit from your classes. Which makes protected attributes part of the public interface.
So, if you want to change the implementation of a class, you can't change how a protected attribute or method works, and what data it stores. If you do, you break all subclasses. Which not only exist in your own source tree, but in your user's source tree too. If you think that changing the implementation will also change protected attributes or methods, make them private.
So if protected methods are essentially part of the public API anyway, why not declare them as public? Why require your user to wear a funny yellow hat (aka declare an inheritance relation) while using your class? There is no good reason.
When seasoned C++ programmers hear that, they often respond with "but if I declare my private data as private a subclass doesn't have access to it! How can it work properly?". The answer is simple: You must provide a sufficiently powerful public API. Your users will be grateful.
One more thing: people often come up with an analogy to the real world, and claim that your kids also have access to your house or flat. This analogy is broken, because in the real world the parents have to take actions to get children (or give somebody the same status as your children). In the programming world, every outsider can write a child class of your classes.
(The reason I posted this in the Perl 6 section of this blog is that Perl 6 has no protected attributes. Which has spawned many interesting discussions, which were the precursor to this blog post.)
Google Summer of Code is a global program that offers student developers summer stipends to write code for various open source software projects. Google Summer of Code 2010 went by quickly, and much was accomplished. The Perl Foundation and Parrot Foundation took part this year, and we were lucky to get proposals from very bright and capable students. We started the summer with 10 students and had 8 students pass their final evaluations. The passing projects include:
Ryan Jendoubi -- Ctypes for Perl Mentor: Reini Urban Blog: http://blogs.perl.org/users/doubi/ Repo: http://gitorious.org/perl-ctypes/perl-ctypes
This project is exciting many Perl developers, because it would minimize the need to use XS, which will make many more pure-Perl modules possible. This improves portability, becaue XS-based modules are notorious for being fragile across operating systems and compiler versions. This adds up to a whole lot of WIN.
Nat Tuck -- Hybrid Threads for Parrot Mentor: Andrew Whitworth Blog: http://parrot.org/blog/836 Repo: http://github.com/parrot/parrot/tree/gsoc_threads
Threads allow a single program to use more than one CPU, which is becoming increasingly important these days. Even mobile phones are multicore! This work aimed at adding threading support to Parrot Virtual Machine. Much was accomplished, but this effort is still on-going. So-called "green threads" were implemented, which is a necessary step to get Hybrid threads working.
Tyler Curtis -- A PAST Optimization Framework for Parrot Mentor: chromatic Blog: http://parrot.org/blog/839 Repo: http://github.com/ekiru/tree-optimization
This project is about providing a framework for optimizing PASTs (Parrot Abstract Syntax Trees). This will be used by language implementors when optimizing their HLLs (High Level Languages). This framework allows all languages on Parrot to benefit from optimizations that are written once, instead of each language implementor writing their own optimizations.
Daniel Arbelo Arrocha -- NFG and single-representation strings for Parrot Mentor: Allison Randal Blog: https://www.parrot.org/darbelo Repo: http://github.com/parrot/parrot/tree/gsoc_nfg
NFG stands for Normal Form Grapheme, and basically means having a standard internal representation of Unicode strings, so that very expensive conversions do not have to repeatedly take place. This makes string-heavy computations much faster and unifies a lot of code.
Carl Masak -- Add support for binary data in Rakudo Mentor: Jonathan Worthington Blog: http://use.perl.org/~masak/journal/ Repo: http://github.com/rakudo/rakudo
Rakudo Perl 6 now supports various binary data formats that were implemented as part of this project. Many relevant tests were also added to the Perl 6 Spec Test Suite as well as improvements and clarifications to the Perl 6 Specification.
Muhd Khairul Syamil Hashim -- Instrumentation Tool for Parrot Mentor: Christoph Otto Blog: http://www.parrot.org/blog/841 Repo: http://github.com/khairulsyamil/parrot-instrument
This instrumentation tool for Parrot allows one to dynamically peek into the execution of Parrot op-codes. This allows for writing profiling tools that can answer questions like "who calls functions X" and "how many Objects of type X were created."
John Harrison -- Improvements to NCI/LLVM Stack Frame Builder for Parrot Mentor: Peter Lobsinger Blog: http://www.parrot.org/ash Repo: http://github.com/ashgti/parrot
This project is a prerequisite for a JIT (Just In Time compilation) framework, which is an important goal for the Parrot community, since Parrot decided that our old JIT subsystem was broken beyond repair and removed it. Parrot has decided to use the very popular LLVM project in our rewrite of our JIT, and this project brings us a step closer on our journey.
Pawel Murias -- Mildew and SMOP on CPAN Mentor: Daniel Ruoso Repo: http://github.com/perl6/mu
This project involved working on Mildew and SMOP. Mildew is a Perl 6 implementation, and SMOP is the library Mildew uses for meta-object programming. You can think of Mildew as a sister to Rakudo Perl 6. Having many implemenations of Perl 6 helps to better define the Perl 6 specification. Updated versions of SMOP and Mildew are now available on CPAN.
The failing projects were:
Justin Hunter -- Rework Catalyst framework instance initialization code Mentor: Florian Ragwitz Mirko Westermeier -- Bulletproofing the Mojolicious test suite Mentor: Marcus Ramberg
Both of these projects passed their midterms, but due to circumstances outside of the program, these students were not able to complete their goals for their final evaluation. Sometimes Real Life throws you a curve ball, like starting a new job, moving to a new city, having a baby and similar things. We wish these students the best of luck, and hope that they complete their projects outside the structure of GSoC.
I am very proud and humbled by all the students and mentors that I worked with this year. I am constantly reminded that there are very intelligent developers that are very young, and The Perl Foundation and Parrot Foundation is very lucky to attract them and have them in our communities. I firmly believe that the passing GSoC 2010 projects have made a large positive impact on our codebases and many people will benefit from them for years to come.
Rock on and keep spreading the Open Source love!
This weekend marks the end of a quite astonishing era. The Pugs repo, that hosted all the amazing Perl 6 activity, is no more. At its height, this repository had 242 committers! I just checked.
The Pugs repository has functioned as a common writing area for Perl 6-related things. First Pugs; then a lot of external modules and scripts written in Perl 6; what would eventually morph into the Perl 6 test suite; even the Perl 6 specification and the standard grammar. Commit bits (write access credentials) were handed out liberally, and newcomers were encouraged to help improve things they found amiss. The degree to which this system worked without a hitch has been quite astonishing. There have been no edit wars, no vandalism, no banning. Just the continuing flow of commits. Trust the anarchy.
So why are we killing off the Pugs repository? Well, technologies come and go; not so much because they worsen by each year, but because our expectations rise in a sort of software inflation. The SVN repository became too high-maintenance, and a transition to git was the natural next step. The Pugs repository has been split up into the git repositories linked to in the previous paragraph. Those bits that don't belong in any of the standard bins remain in the Mu repository. (Its name is a reference to the most-general object type in Perl 6, what in other languages is commonly known as 'Object'.)
I for one salute our new git-based overlords! May the commits keep flowing even under this new system. Also, moritz++ for carrying out the move.
Out on Reddit, my last post got some interesting comments. In particular, one commenter creates a finite series of squares, and then asks “That’s all well and good, but what would I have to do to say ‘up to 900′ instead of saying ‘up to 30^2′?†That’s a great question, and I’d like to explore the answer to it, though I’m going to stick with the Fibonacci series for my examples.
First, there’s a very quick and dirty answer to this. If you modify the original series, it is easy to add a limit term:
> my @Fibonacci := (0, 1, -> $a, $b { $a + $b } ... 900);
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610
That gives you all the Fibonacci numbers less than or equal to 900. But the list is no longer expandable; with this one, if you say @Fibonacci[100], rather than calculating the 100th value of the series, it will just act like it’s a normal out-of-bounds array access.
In an ideal world, we’d have an easy way of looking at just part of an infinite series. What tools do we have in our toolbox? Well, there’s grep:
> my @Fibonacci := (0, 1, -> $a, $b { $a + $b } ... *); 1;
1
> my @F-low := @Fibonacci.grep(* < 900); 1;
1
> say @F-low[10]
55
> say @F-low[20]
…and that last one is where the trouble starts, because it never comes back. By 20, we’re already well over 900. But grep’s been fed an infinite list, and it doesn’t know anything about it, so it just keeps on looking forever. So that won’t do.
The next tool we have is the first method.
> my @Fibonacci := (0, 1, -> $a, $b { $a + $b } ... *); 1;
1
> say @Fibonacci.first(* <= 900)
0
> say @Fibonacci.first(* > 900)
987
As you can see, first is great at finding the first Fibonacci number greater than 900. It’s pretty useless for finding all the numbers less then 900, though.
So, the bad news here is that, as I write this, Perl 6 does not have a method which can give us a finite portion of an infinite list based on some test, other than “give us the first N elementsâ€. The good news is that the function we want is dead simple to write:
sub take-while(@a, Mu $test) {
gather {
for @a.list {
last unless $_ ~~ $test;
take $_;
}
}
}
This is actually a straightforward modification of the normal grep code. $test is defined as Mu so it can be anything that can be smartmatched against, including junctions. Then we scan the list, exiting if a smartmatch against the current element fails, and otherwise returning the element in our output list.
This is just the trick for the problem in question:
> my @Fibonacci := (0, 1, -> $a, $b { $a + $b } ... *); 1;
1
> take-while(@Fibonacci, * < 900)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610
> take-while(@Fibonacci, * < 400000)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765 10946 17711 28657 46368 75025 121393 196418 317811
> take-while(@Fibonacci.grep(* %% 2), * < 400000)
0 2 8 34 144 610 2584 10946 46368 196418
> take-while(@Fibonacci, Int)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765 10946 17711 28657 46368 75025 121393 196418 317811 514229 832040 1346269 2178309 3524578 5702887 9227465 14930352 24157817 39088169 63245986 102334155 165580141 267914296 433494437 701408733 1134903170 1836311903
So here we have it: we’ve still got the efficiency of the de facto memoized solution, but we can also carve off portions of the list based on some criteria. I threw in the last one there just to show it didn’t have to be a Code block passed in; in this case, it shows us how far we get down the series with Int values before our current limited integer math throws in the towel and gives you back a Num instead. (Note that once we get arbitrarily large Ints in Rakudo, that line will no longer work, because take-while(@Fibonacci, Int) will return an infinite list!)
And now for the even slightly better news: as of today, take-while is available in the List::Utils module.
It is with a peevish exultation of spirit that I announce on behalf ofthe Yapsi development team the September 2010 release of Yapsi -- soon to be a major motion picture -- a Perl 6 compiler written in Perl 6.
You can download it here (or, if you happen to be on an avian-carrier-based network, you can "pidgeon" it here).
Yapsi is implemented in Perl 6. It thus requires a Perl 6 implementation to build and run. This release of Yapsi has been confirmed to work on both Rakudo Star 2010.08 and Rakudo Star 2010.07.
Yapsi is an "official and complete" implementation of Perl 6. Yapsi's official status has been publicly confirmed by Patrick Michaud, the Rakudo pumking. The claim about Yapsi being complete... well, it might just be what PR people sometimes refer to as "a slight exaggeration". On the bright side, it's becoming less so with each new release.
This month's release brings you unless and until, as well as our-scoped variables:
$ yapsi -e 'my $a = 0; unless $a { say 42 }'
42
$ yapsi -e 'my $a = 0; until $a { say ++$a }'
1
$ yapsi -e 'our $a = 42; { my $a = 5; { say our $a } }'
42
For a complete list of changes, see doc/ChangeLog.
Quite a lot of features are within reach of people who are interested in hacking on Yapsi. See the doc/LOLHALP file for a list of 'em. In fact, that's how isBEKaml++ implemented 'unless' this month. (After which he exclaimed "that was easy!" and tackled 'until'.) If you're wondering whether you're qualified to help with the Yapsi project, that probably means you are.
Yapsi consists of a compiler and a runtime. The compiler generates instruction code which the runtime then interprets. In Yapsi, that instruction code (unfortunately) is called SIC[!]. Until further notice, SIC as a format changes with each monthly release for various, mostly good reasons. However, if you write a downstream tool that makes assumptions about the SIC format, someone might change it just out of spite. SIC is explicitly not compatible with later, earlier, or present versions of itself.
An overarching goal for making a Perl 6 compiler-and-runtime is to use it as a server for various other projects, which will hook in at different steps:
Another overarching goal is to optimize for fun while learning about parsers, compilers, and runtimes.
Have the appropriate amount of fun!
Masak recently posted on the series operator solution to Pascal’s Triangle. I’d like to focus on an important property of his final solution that hasn’t been much discussed yet.
But first, let me back up and consider another well-known problem with similar properties: the Fibonacci series. A straightforward recursive solution looks like this in Perl 6:
sub Fibonacci($n) {
return $n if $n == 0 | 1;
Fibonacci($n - 1) + Fibonacci($n - 2);
}
This nicely implements the definition, but it has one critical problem — it is incredibly inefficient. (And not just in Rakudo — in any language!) The reason is that calculating Fibonacci(N) requires all the work of calculating Fibonacci(N-1) and Fibonacci(N-2). The number of subroutine calls involved grows as fast as the series does.
The standard recursive technique for dealing with this problem is called memoization. Basically, you cache the values which have already been computed somewhere, so you only need to compute them once. That’s a bit awkward to implement by hand — it roughly doubles the complexity of the routine, I’d say. Luckily, lots of languages have automatic ways to apply memoization. Perl 6 is supposed to, with the “is cached†modifier to a subroutine definition, but Rakudo doesn’t have it implemented yet.
That’s okay, because there is more than one way to do it, and there is a great way which is available already in Rakudo.
> my @Fibonacci := (0, 1, -> $a, $b { $a + $b } ... *); 1;
1
> say @Fibonacci[5]
5
> say @Fibonacci[10]
55
> say @Fibonacci[30]
832040
(I’ve used parentheses instead of do because that’s how I roll, but otherwise this is exactly the same idea as Masak’s solution. The additional 1; is needed only because I typed this into the REPL, and without it the REPL will try to evaluate the entire infinite series so it can print it out.)
You can’t see it here, of course, but calculating these numbers was very quick. That’s because this solution using the series operator is equivalent to writing a very smart memoization. Infinite lists in Perl 6 are implemented (roughly speaking) with an array of values that have been already calculated, and an iterator which knows how to calculate further values as needed.
So when you say @Fibonacci[5] the first time, the list calls the iterator generated by the series operator to get the first six values, storing them in @Fibonacci[0] through @Fibonacci[5]. (Those values are generated in a straightforward iterative way, so it is very efficient.) When you then say @Fibonacci[10] those first six values are still there, and only the next five values must be calculated. If at that point you said @Fibonacci[8], that is just a simple array lookup of the already calculated value!
Think about it. It would take a very smart automatic memoizer to somehow figure out that the function would only be called when $n was a non-negative integer, so that an array could be used to efficiently cache the results. Using a series operator this way gets you that kind of performance automatically already in Rakudo.
So it’s a double win. Using the series operator is not only the most succinct way to express this series in Perl 6. It’s also an extremely efficient way of calculating the series.
So, I wrote a program to generate Pascal's triangle. The first ten rows of the triangle, at least. It only used simple features of Perl 6, such as scalars, nested arrays, and for loops.
my $ELEMENTS = 10;
my @pascal = [1];
for 1 .. $ELEMENTS - 1 {
my @last = @pascal[ * - 1 ].list;
my @current;
push @current, @last[0];
for 0 .. @last - 2 {
push @current, @last[$_] + @last[$_ + 1];
}
push @current, @last[ * - 1 ];
push @pascal, [@current];
}
say @pascal.perl;
In fact, save for simple mechanically substitutable differences, it could have been a Perl 5 script. In fact, with a bit of manual array allocation, it could have been a C script. That's OK; there's a tolerance in the Perl community of writing code that looks like it was thunk in some other language.
But I've heard that Perl 6 is great at doing things with operators. For example, the Z operator, which interleaves two lists, seems to be able to help me write my push statements more succinctly:
my $ELEMENTS = 10;
my @pascal = [1];
for 1 .. $ELEMENTS - 1 {
my @last = @pascal[ * - 1 ].list;
my @current;
for (0, @last) Z (@last, 0) -> $left, $right {
push @current, $left + $right;
}
push @pascal, [@current];
}
say @pascal.perl;
The parentheses before and after the infix:<Z> aren't necessary, because the Z operator has looser precedence than comma. They're just shown here to make your eyes accustomed to reading this construct.
In fact, now that only the addition is performed in the inner loop, I might as well use the Z+ operator, which does this for me.
my $ELEMENTS = 10;
my @pascal = [1];
for 1 .. $ELEMENTS - 1 {
my @last = @pascal[ * - 1 ].list;
my @current = 0, @last Z+ @last, 0;
push @pascal, [@current];
}
say @pascal.perl;
Now as the remaining loop shrinks to a size I can take in all at once, I see a bit more clearly what I'm doing: I'm building each new list from the previous one. I could feed the previous list into a named function to get the current one:
my $ELEMENTS = 10;
my @pascal = [1];
sub next-list(@p) {
[0, @p Z+ @p, 0]
}
for 1 .. $ELEMENTS - 1 {
my @last = @pascal[ * - 1 ].list;
my @current = next-list(@last);
push @pascal, @current;
}
say @pascal.perl;
Or I could just feed it into a in-place anonymous sub.
my $ELEMENTS = 10;
my @pascal = [1];
for 1 .. $ELEMENTS - 1 {
my @last = @pascal[ * - 1 ].list;
push @pascal, (sub (@p) { [0, @p Z+ @p, 0] }).(@last);
}
say @pascal.perl;
But why even a sub? Perl 6 has a lighter construct, namely a "pointy block" (also known as a "closure" or a "lambda"). It doesn't participate in the call stack, and it's slightly easier to write.
my $ELEMENTS = 10;
my @pascal = [1];
for 1 .. $ELEMENTS - 1 {
my @last = @pascal[ * - 1 ].list;
push @pascal, (-> @p { [0, @p Z+ @p, 0] }).(@last);
}
say @pascal.perl;
Let's look at what the code does. Seed with one element. Calculate the next element based on the previous one. Stop at some point.
But that's exactly what the series operator does. The one that's written with three dots. We have a starting value, a way to get from one value to the next (our code block above), and a stopping value.
Well actually, we don't have the stopping value. But that's OK, since the series operator is lazy. So if we only request the first 10 values, it won't loop forever giving us the rest of the list.
my @pascal := do [1], -> @p { [0, @p Z+ @p, 0] } ... *;
say @pascal[^10].perl;
(The extra do required because of a shortcoming in Rakudo.)
Now. Something very much like this code was posted first on Rosetta code and then on Moritz' blog. (TimToady used a sub, but said later that he'd have preferred binding.)
A couple of Perl 5 people's reactions were — somewhat uncharacteristically — of a negative flavour, similar to how people seem to react to the periodic table of operators:
@shadowcat_mst: an excellent example of why I consider camelia perl to be a language research project more than a production language
@pedromelo: I'm seriously considering this post as an example of what I don't want Perl6 to become...
I think these reactions are mainly feature shock. Higher-order operators, pointy blocks, and the series operator... they're all good, well-established features, which find daily use in Perl 6 programs. Maybe using them all together like that flung some people off the deep end. Never mind that the resulting script is all essential complexity, with virtually no boilerplate from the original script left.
This is the first time that's happened. I think it's important to listen to what Perl 5 people think and to try to respond to that. But I also think that this time, it's a case of them seeing some highly idiomatic Perl 6, and freaking out a bit.
And I think that that, in some odd sense, is a good thing. Well, not freaking people out, per se. But the fact that we did shows that there's something forming which might be tentatively called "idiomatic Perl 6": people on the inside can read it quite easily, but those on the outside, even Perl 5 folks looking in, instinctively go "eeeeew!".
That's OK. You're not meant to start with the idiomatic stuff. Language acquisition takes place step by step, and that goes for learning Perl 6 as well. On the way there, just don't confuse distaste with lack of familiarity.
Rakudo has been leaking memory for a few month. The other day, after some nagging, Will Coleda identified a memory leak, and Tyler Curtis fixed it.
Now we can again make long-running processes with Rakudo. For example for my talk at YAPC::EU I plotted a resonance curve. For that I needed to start a new Rakudo process for every data point because it would leak so badly that it died after processing a few data points. Now I recalculated a whole curve in one process, with memory usage not exceeding 200MB of virtual mem.
I also had some fun recalculating a mandelbrot fractal in a size that would previously make Rakudo segfault or consume too much memory.
(Rendered with colomon's mandelbrot code)).
SF’s ABC project continues to be dead in the water, but that hasn’t stopped me from needing some quick ABC scripts. I’m preparing a mini-book of tunes for an Irish Traditional Dance Music workshop, and needed a couple quick programming tricks to make my life easier.
First, a file of ABC tunes should have a unique “X:†index field for each tune. I’d gathered these tunes from around the Internet; correspondingly, about half had “X: 1″, and the rest had some random number. This was easily fixed with a simple Perl 6 script:
my $x = 1;
for $*IN.lines {
when /^X\:/ { say "X: { $x++ }"; }
say $_;
}
The only thing at all odd here is that I’ve used when in a for loop. It just smartmatches the topic ($_) against the regex there, and if it’s a match, executes the code that follows and then skips to the next iteration of the loop.
The trickier thing I needed to do was transpose a couple of tunes, because the versions I found on the Internet were in the wrong key. Cutely, this is pretty easily done with the .trans method in Perl 6.
for $*IN.lines -> $line {
# say (~$line).trans("C..GABc..gab" => "FGABc..gab"); # from D to G
say (~$line).trans("C..GABc..gab" => "D..GABc..gabh"); # from G to A
}
This needs to be applied only to the notes of the tune, and if you use the “G to A†transposition and you have an “h†as a note, you need to switch it to “c’â€. (You also need to change the declared key signature in the tune’s header. And accidentals might need work. This is definitely crude ABC-handling code, but it suited my purposes perfectly last night.)
There is one big gotcha here. Why is it (~$line) instead of just $line? Well, internally .lines is calling .chomp. And apparently .chomp no longer returns a Rakudo Str, which seems to mean some other .trans is executed, causing no end of problems.
Anyway, at some point I intend to try to get the real ABC module working in Rakudo (if no one else beats me to it). In the meantime, I’ve got to survive a weekend’s worth of folk festival first.
Today on IRC, Larry Wall showed this piece of Perl 6 code, which he wrote for Rosetta Code:
sub pascal { [1], -> @p { [0, @p Z+ @p, 0] } ... * }; say pascal[^10].perl # output (reformatted for easy readbility): # ([1], # [1, 1], # [1, 2, 1], # [1, 3, 3, 1], # [1, 4, 6, 4, 1], # [1, 5, 10, 10, 5, 1], # [1, 6, 15, 20, 15, 6, 1], # [1, 7, 21, 35, 35, 21, 7, 1], # [1, 8, 28, 56, 70, 56, 28, 8, 1], # [1, 9, 36, 84, 126, 126, 84, 36, 9, 1])
That's Pascal's triangle, generated in one line of Perl 6.
The ... is the series operator, which generates lists by
feeding the previous value(s) (here always one array) to the generating
block on its left, until it reaches the goal on the right (in this case
"whatever", which means it returns a lazy, infinite list).
So for example if the previous item was the array [1, 2, 1],
the code block evaluates 0, 1, 2, 1 Z+ 1, 2, 1, 0.
Z is the zip operator, Z+ is pairwise addition
(ie adding the pairs that the zip operator produced). In our example that
leads to 0+1, 1+2, 2+1, 1+0 or 1, 3, 3, 1.
It takes a while to get used to the meta operators and the series operator, but once you've understood them, you can do pretty neat things with them.
Disclamer: I don’t want to provoke another instance of something I’d be writing about, so I’m going to refer to languages as to meals.
We all like eating. Some of us prefer only one or two kinds of food, but most of us have experiences with many types, yet there are this few ones which we prefer. But more often than I’d like, we yell at each other that our types of food are better, instead of serving our favorite meals with happiness on our faces. Or we insult each other because of the type of food someone else prefers.
Isn’t it sick?
I’m not a professional cook, but I tend to wander there and around in the cooking community. Let me tell you a few stories I experienced in the recent time.
So there’s a cooking convention, and a two cooks are showing some stuff they cooked using Mushrooms. At the beginning, they start to exaplain, why Mushrooms? “Oh you know, we like Mushrooms, and would any of you like to cook these using Pasta? Are you willing to punch yourself in the face after cooking Pasta? And if you enjoy using the cookbooks about Pasta, you must be masochists!â€. I’m glad I was watching it recorded rather than live, as I might have looked stupid standing up and leaving the room. But really, this were some serious cooks, why do they yell at other types of food instead of showing why their are nice? Seems not very professional, and shows Mushroom lovers in a bad light.
But the sad thing is, many of them are like this. On the cooking reddit (sometimes written as “cookitâ€), in every article or news headling about a new meal with Pasta, dozens of Mushroom lovers appear yelling how madly they hate Pasta. Really, comments are filled with this cretinism. They shout at how Pasta is old-fashioned, difficult to eat and how it looks bad. Then why do they need to tell this to everyone, while they see how they keep being downvoted? I asked one of the redditors, why so much hate against Pasta? “It’s not hate, it’s disgustâ€, he replied. Hey, stupid! No need to spoil somebody else’s taste, why don’t you cook something good with your beloved Mushrooms instead of shouting at how madly you hate Pasta? Sadly, they seem to attack Tomato lovers too, everytime telling everyone how Mushrooms are better for everything and everyone should quit cooking Pasta and Tomatoes and move to Mushrooms instead. I once even saw something horrifying on the Mushrooms mailing list, although I don’t usually read those. There was a quote: “and what defines a Mushroom activist anyway? Blowing up Pasta meals worldwide?†What on earth is wrong with you people?
Luckily, not everyone is like this. I tend to see friendly Mushroom lovers, which sometimes cook Pasta too, although I usually see them on Pasta related IRC channels. Good to know there are good people in this community. Good to know they’re cooperating with us, there are so many wonderful things we can cook together. For example, do you even know the awesome things you can cook with exotic animals, like Parrots?
Now, you may have alredy guessed what kind of food is which language in this story. But there’s a reason I didn’t write a cheatsheet anywhere. It could be related to any of you, probably to me also. Just try to think about from a wider perspective. Hating other people’s food does not make you a better cook. It’s not nice either, when someone jugdes you basing on the type of food you cook, or eat? Shoving your opinions down peoples’ throats is always bad.
From Wikipedia, the free encyclopedia:
In game theory and economic theory, zero-sum describes a situation in which a participant's gain or loss is exactly balanced by the losses or gains of the other participant(s). If the total gains of the participants are added up, and the total losses are subtracted, they will sum to zero.
Being advocate, implementor, tester and co-designer of a new programming language, I often hear objections along the lines of you are killing $other_programming_language, combined with a mixture of fear and resentment. People are afraid that having a new player on the market will decrease market share of their own, favorite programming language.
While I can understand these thinking patterns, there is no reason for concern. The market for programming languages is not a zero-sum situation. While I don't have hard data, I have the impression that the programming job sector is growing, and the US government expects it to grow further, too.
Certainly the growth of world population sets a rapidly increasing baseline, and even if we assume a constant percentage of all people related to programming in some way, the total number of programmers rises, and will continue for quite some time.
(I'm pointing to some resources about programming jobs, and I fully realize that it's not the same as number of overall programmers; but it's easier to get data for jobs, and I do think that the general trend statements are true for both).
So as long as the total number of programmers increases, a decrease in relative market share doesn't automatically mean a loss. In fact the job trends show an increase for "scripting" languages, and while Ruby is certainly the winner in terms of growth, Python, Perl and PHP win too!
Non-job data shows for example a noisy but steady growth of uploads to the Comprehensive Perl Archive Network (CPAN) -- data from a programming language that is often perceived as a loser of ruby's and python's success.
A recent Linux distribution trend analysis fell into the same trap: it shows relative numbers of search terms, and talks about a decline for all distributions except Ubuntu. Again I don't have hard numbers (the mirror infrastructure of most Linux distributions makes it nearly impossible to get accurate download counts), but I haven't seen any evidence that total usage numbers of any of the Linux distributions actually decreased.
(This post isn't very punny. For those of you who need puns to survive, try to figure out why jnthn++ named the IRC logs "the hottest footwear" recently. The answer, as with all good puns, is highly unsatisfying.)
My quest for a Perl 6 implementation takes me ever deeper into the esoterics of lexpads, runtimes, and a far-more-than-everything-you-needed-to-know mindset. Today some random firings in my brain turned into the following conversation on #perl6.
During the conversation, I proposed two theories, both of which turned out to be wrong. (pmichaud++ shone the necessary light both times.) Being wrong felt less important than getting my mental model fixed.
I first thought of presenting the results of the below conversation as a simple tutorial ("How our declarations work. The complete guide."), but now I think that the conversation, minimally edited, manages to be such a tutorial on its own.
Besides, blogging things in their raw and undigested form is a sign of the times. Enjoy!
<masak> I have a question. is there a need for a special "package lexpad" containing 'our'-declared variables, or can the package lexpad simply be equated to the topmost lexpad in the package?
<masak> my suspicion is the latter, but I might be missing something.
<pmichaud> the package lexpad can't be the same as the top most lexical
<pmichaud> module XYZ { my sub abc() { ... } }; # abc should not appear in the package
<masak> oh!
<masak> right.
<masak> so, separate one, then.
<jnthn> Additionally, lexpads are meant to be static by the time we hit runtime, and you're allowed to shove stuff into the package dynamically. Not quite sure how those two hold together.
<pmichaud> well, module XYZ { ... } creates a lexical XYZ entry that holds the package entries
<jnthn> Aha!
<pmichaud> and it's just a hash, really.
<masak> does inserting the package lexpad below the outside lexpad (and above the topmost lexpad) make sense? that way, Yapsi wouldn't need any special opcodes for doing 'our'-variable lookups.
<pmichaud> the package lexpad is an entry in the outside lexpad, yes.
<pmichaud> I'm not sure it encapsulates the nested lexpad, though.
<masak> hm.
<masak> if it doesn't, I don't really see how it's visible from inside the package.
<masak> I've more or less convinced myself that sandwiching it between outer and topmost is what I want to do for Yapsi.
<pmichaud> our &xyz can make an entry in both the package and in the lexical.
<pmichaud> this is what rakudo does now.
<pmichaud> we have to do similar things for methods already, too.
<masak> sure. it makes entries in both.
<pmichaud> by having entries in both, that's how it's visible inside the package
<masak> hm, indeed.
<masak> no need to have the package lexpad visible from inside.
<pmichaud> anyway, sandwiching might work too. haven't quite gotten to that point in Rakudo thinking yet. And it can get a bit tricky with multis.
<masak> no need to sandwich it in, either. it can sit in limbo outside the tree of scopes.
<pmichaud> oh, I know why it perhaps shouldn't (or should) be visible:
<pmichaud> my $x = 'lexical'; module XYZ { say $x; { our $x = 'package'; } }
<masak> ...yes?
<pmichaud> I'm pretty sure "say $x" needs to grab the 'lexical' $x, not the one that might be "sandwiched" in a package.
<masak> of course.
<masak> that falls out from ordinary scope nesting and shadowing.
<masak> innermost block binds its lexical to the container in the package lexpad.
<masak> so, that speaks out against sandwiching.
<masak> pmichaud++
So there you go. There's a separate package scope, and it isn't sandwiched.
(Answer: The missing link is the "IR clogs" meme from #parrot. I can hear you groaning... I did warn you.)
A little while ago, I submitted a Hague Grant application to do a significant meta-model overhaul for Rakudo. I’m hopeful it’ll be approved soon, but lack of approval so far has, of course, not stopped me spending time considering the problem space, and even engaging in a little prototyping. I’ve got a lot of posts planned for the coming weeks and months about this work. In this one, I want to take some time to discuss some of the aims of this work and give you an idea of the roadmap.
I’ve been hacking on Rakudo for the best part of two and a half years now. It goes without saying that I – and others – have learned an enormous amount about Perl 6′s needs in that time. Perl 6 is a very feature rich language, but it’s built upon a rather smaller set of primitives. As I set about re-designing and re-implementing a couple of those, I constantly have to ask myself, “OK, can this design handle everything we want to put on top of it?†Of course, experiences from Perl 6 implementations – Rakudo and others – haven’t been and won’t be my only sources of ideas. There’s plenty of other object model implementations to look at, including Moose, CLOS and Smalltalk. It’s also worth looking at what languages like Java and C# do in this area – let’s face it, they’re OO languages that are known to run pretty fast atop of their respective VMs, so it’s good to understand how they achieve that. And then there’s all of the various papers that academia has churned out, which can also provide a great deal of inspiration, or at least exploration of problem spaces.
So, what do I actually want out of this process?
A really big ticket item is that I want a meta-model that supports gradual typing. It may seem odd to put this at the top of the list, but it’s actually one of the things that I think will play out as being really important, and I’m not sure it’s received enough attention yet in Perl 6 – at least, not in the meta-model design space. We often talk about Perl 6 as being a dynamic language. Well, it is – but it makes more stuff statically knowable than you might first imagine. The possibility for a programmer to write type annotations means we may be given a lot of information to work with, but even knowing that everything descends from Mu gives us some optimization opportunities. Most parameters are actually of type Any, which gives us more information to work with.
Today in Rakudo, writing type annotations may make your program slower rather than faster, even though you give more information. I want to fix that. Putting it another way, this work is about producing a meta-model that deeply supports both static and dynamic typing, with the ability to give us gradually greater amounts of optimization and safety as we move gradually towards the static end of the scale. (Some people at this point may ask about type inference and wonder if we can do some optimizations to dynamic programs using that; the answer is yes, we may well be able to do that and it’ll be some interesting future work, but having a way to put the calculated types to use is a prerequisite.)
This directly ties into performance, which is one of Rakudo’s biggest issues at the moment. We need improvements in terms of compile time performance and runtime performance. Various things need to get faster, notably:
Perl 6 is built out of method dispatches. While the programmer may not give us that much information to work with so far as types go, that need not be the case in the internals. In fact, if we want to write a lot of Perl 6 in Perl 6 and have it performant, we’re going to have to do things that give the compiler and runtime enough information to generate fast code. Of course, when we are in a dynamic situation, we need to be able to do runtime type checks much more quickly than we can today too.
Then there’s memory consumption. Consider a Rat (rational number type) today. It contains two attributes holding Ints, which in term constitute two PMCs each (a Parrot Integer PMC that holds the value and the Perl 6 Int subclass of it). Worse, the Object PMC uses a ResizablePMCArray for attribute storage. I’ve probably lost many of you in Parrot guts terminology by now, but the bottom line is that one Rat today can carry around 8 garbage-collectable objects! Well, we made it work – but this clearly has gotta change if we want to make it fast. 3 objects is perhaps far more reasonable – a Rat object with two Int objects within them. (It goes without saying that the delegation we end up doing here as well as the extra GC overhead is also a source of speed woes.)
This leads me nicely to native types, and thus representations. The SMOP project has done a lot of work looking into the area of representation polymorphism – something that Rakudo has largely ignored up until now. However, now we want to do native types it’s time to dig in and take this issue seriously. This especially comes into play in the area of boxing and unboxing – a task we’ve mostly punted on and let Parrot worry about. Today we often avoid boxing things into Rakudo-level objects when we don’t have to, because it’s so much slower than boxing to Parrot PMCs. This leads to other bits of cheating and – now and then – the cheating leaks through to user space and causes trouble. We should be able to box natives up fast enough that we feel no need to cheat and lie. These cheats and lies also carry a runtime cost.
One other area that Rakudo does badly at today is startup time. We don’t need an incremental improvement here, we need an order of magnitude one. At startup today, we essentially build up the entire runtime library ecosystem from scratch (apart from some things we can lazily defer until later). Even if we can get faster at building that, it’d be faster still to be able to just build it once, freeze (aka serialize) it, and then just have to quickly thaw (aka deserialize) it at startup and be ready to run. Parrot today does have some freeze/thaw support, but it’s not enough. Notably, it doesn’t handle the issue of linkage – that is, having instances of a type defined in library A serialized in library B and matching them up. So I’ll be looking at how we can extend that.
Going even further, I want us to actually build the meta-objects at *compile time*. Not just at compile time, but actually in action methods (so that by the time we’re done with the parse, we have them, and just have to fudge in the compiled Parrot Sub objects into the various slots. Why? Because that way we have the meta-objects around to provide us with the information we need to do gradual typing and a host of other optimizations and error checking. We could, of course, maintain a parallel set of compile-time and run-time versions of the information. We partially do that today, but carrying around two copies of the same information is just asking for them to get out of sync. Not to mention that we want to serialize them anyway, so we need to build them at some point during the compile. Put another way, I want to unify our compile time and runtime meta-models. I’ll write in a lot more detail on this later on – it represents something of a sea-change to our compilation model for packages.
Portability is another concern. Today Rakudo only runs on Parrot, but for some time now the Rakudo team have been talking about running on other backends too. So far, getting something out that works – in the form of Rakudo * – has been more important than that, and it goes without saying that I’ll be delivering an implementation of the meta-model and all the bits around it that runs on Parrot. However, there’s quite a bit of interest in having Rakudo working on additional backends, and I’m very keen that whatever meta-model design Rakudo ends up with, it is cleanly implementable on Parrot, the JVM, the .Net CLR, LLVM and [your runtime here].
There’s a lot of things to juggle here, and there’s going to be a lot of changes landing in the coming months. In these early days, I’ll be mostly prototyping. Once I’m happy with the overall design, I’ll dig in to implementing it on top of Parrot, and modifying nqp-rx. That will happen in a branch. Once that is done, I’ll branch Rakudo and do similar there. At some point, as we did when implementing the new grammar engine, those branches will become the mainline.
I *do* expect this to be a lot less painful on the Rakudo side than what we went through with the infamous “ng†branch. On the nqp-rx side that’s going to be a bit harder to promise; I will of course not break anything for the sake of breaking things, and for people writing compilers using PCT I really hope this will be close to seamless. People writing NQP code in expectation that it’s going to produce a certain bit of PIR, on the other hand, may be in for more of a surprise. As an overview (subject to change):
It was just under a year ago when pmichaud++ dug into the nqp-rx effort, which was a prerequisite for bringing Rakudo up to the level where we could reach the Rakudo Star release series. We made it – Rakudo may not be fast today, and for sure there’s bugs, but we successfully put out a release that delivered on many of the feature promises of Perl 6. This batch of work is a prerequisite for taking us to the next level: towards one where we run faster, have a path to implement features and optimizations that we can’t today, where the meta-programmer can work productively, and where more stuff Just Works the way it should.
Let’s have some -Ofun. :-)
So, the last two posts suggest there is major overhead to using the lazy iterator approach with .lines. I decided to explore this by rolling my own iterator to read a file. First, I suspected the gather / take has a big overhead, so I just tried for a basic customer iterator first:
class LinesIter is Iterator {
has $!filehandle;
has $!value;
method infinite() { False } # or should this be True?
method reify() {
unless $!value.defined {
$!value := pir::new('Parcel');
my $line = $!filehandle.get;
if $line.defined {
pir::push($!value, $line);
pir::push($!value, LinesIter.new(:filehandle($!filehandle)));
}
}
$!value;
}
}
That takes 21.7, very slightly more than the standard gather / take version. So much for the theory gather / take is inefficient here!
I’m told that the spec requires that .lines be strictly lazy so you can mix in calls to .get. I don’t know where, and it seems a bit crazy to me. But anyway, by those lights the following potential optimizations are actually illegal, because they break the strict connection between .lines and .get.
Here’s one that does three .gets at a time, cutting the number of iterator objects created by two-thirds.
class LinesIter is Iterator {
has $!filehandle;
has $!value;
method infinite() { False }
method reify() {
unless $!value.defined {
$!value := pir::new('Parcel');
my $line = $!filehandle.get;
if $line.defined {
pir::push($!value, $line);
$line = $!filehandle.get;
if $line.defined {
pir::push($!value, $line);
$line = $!filehandle.get;
if $line.defined {
pir::push($!value, $line);
pir::push($!value, LinesIter.new(:filehandle($!filehandle)));
}
}
}
}
$!value;
}
}
Clocking in at 15.6 seconds — significantly better than the current .lines implementation, significantly worse than the .get version — this was actually the best variant I came up with.
I tried upping the count to 8, but it actually ran a touch slower then. And jnthn suggested a version which tried to optimize creation of the iterator objects, but it actually ran significantly slower than the naive version.
I’m not really sure where this leaves us…
Overnight pmichaud rewrote .chomp in PIR to optimize it, leading to huge improvements in performance. The .lines version of the code has gone from 55s to 21.5s; the .get version from 44s to 7.2s. Weirdly enough, the .slurp version has gone from 5.6s to 9s. (No idea why.)
The good news is that the performance of the .get version is clearly much, much better. (Still with plenty of room for improvement!) The bad news is that the penalty for using the iterator-based version is now much more significant — it adds nearly 200% to the read time!
But then, that’s another potential spot for optimization…
I’ve been worried about the performance of .lines reading text files due to iterator overhead, so I decided to time it tonight. It is a problem, but it appears to not be the biggest problem!
As usual, I probably did things backwards. I started off by creating a ten thousand line text file from Violet Jacob poems and Rakudo’s core.pm. Then I ran it against this:
my $count = 0;
for $*IN.lines {
$count++;
}
say :$count.perl;
That takes 55s on my MacBook Pro.
The .lines method is just an iterator wrapper around .get, so I tried a version which calls .get directly next.
my $count = 0;
loop {
$*IN.get // last;
$count++;
}
say :$count.perl;
That takes 44s on the MBP. So using the iterator adds 25% to the execution time. That’s bad, but in a certain sense, less bad than the straight .get version being so incredibly slow.
Next attempt, .slurp. What’s the overhead of doing things line-by-line, with an autochomp?
$*IN.slurp;
That takes 5.6 seconds. So the line-by-line overhead is terrible AND even the slurp version is crazy slow.
jnthn coded up two PIR versions for comparison. Version 1 reads the entire file in PIR — that took 0.84 seconds on my MBP. Version 2 reads the file line-by-line in PIR, and is drastically faster — 0.03 seconds. (pmichaud++ reported this discrepancy to the Parrot team.)
It looks like .chomp might be the best point of attack. It looks grotesquely inefficient at the moment. But it’s time for bed now….
Update: see the next post for reports on how pmichaud’s .chomp optimization (mentioned in the comments) improved the situation.
During the last one and a half month, I've been working on making regex modifiers easily available in Rakudo.
The regex compiler itself has to support only a few of the adverbs that can be applied to regexes; those include :ignorecase, :sigspace, :ignoremark and :continue/:pos. NQP-rx, the regex engine that Rakudo uses under the hood, supports those (except :ignoremark), so previously you could write
if 'ABC' ~~ /:i abc/ { say "case insensitive match"; }
But not
if 'ABC' ~~ rx:i/abc/ { say "case insensitive match"; }
nor m:i/abc/, for that matter.
I've patched Rakudo to actually recognize those adverbs outside of the
regex, and also for s/// substitutions.
Another category of adverbs are those that apply to regex calls, not to the
compilation of a regex. Among those are :global/:g, :overlap/:ov, :nth($n),
:x. I've implemented those for substitutions, but implementing them for
m// turns out to be quite a bit harder.
The reason is the return value: each regex match returns a Match object, which can store positional and named parts. S05 says that regex matches with multiple results should return a single match object, with all results as positional parts. It can be distinguished from a normal match object by evaluating it in slice context... which Rakudo doesn't support yet.
Now the subst method and thus s/// are
implemented by calling .match(:global, ...), and without slice
context, it can't distinguish between multiple matches, and a single match
with subcaptures. And so my changes to the global match broke the
substitution, and I see no easy way to fix it.
Anyway, here are a few examples of what works today:
$_ = 'ab12fg34'; s:g/\d/X/; .say; # output: abXXfgXX $_ = 'Hello, World'; # :ii is the same as :samecase s:ii/world/perl/; .say; # output: Hello, Perl $_ = 'I did not know that that work together'; s:2nd/that/they/; .say; # output: I did not know that they work together
This time together with a Perl 6 screencast to show the thought operator and the +- operator of Perl 6. Direct link to the Perl 6 subroutines screencast
I just tried the thought operator in Perl 6. Typed in this:
"szabgab" .oO "Perl 6 is cool";
and got the following:
szabgab thinks Perl 6 is cool
I also tried the +- operator:
2.78 ~~ 10 +- 3 ?? 't' !! 'f' # f 7.5 ~~ 10 +- 3 ?? 't' !! 'f' # t 13 ~~ 10 +- 3 ?? 't' !! 'f' # t 13.1 ~~ 10 +- 3 ?? 't' !! 'f' # f
I think you get the point here. We check if the value on the left is between 10 +- 3.
~~ is the smart match operator, ?? !! are the two parts of the ternary operator of Perl 6 and +- is, well not everyone has the +- or the thought operator but if you read on you'll see how you can create one for yourself.
Subroutines in Perl 6 are declared using the sub keyword just as in Perl 5 But in Perl 6, after the name of the subroutine one can provide a signature, describing the parameters that function accepts.
use v6;
sub f($a, $b) {
return "$a - $b";
}
say f(4, 2);
If called with a different number of arguments then Perl will issue an error message:
say f(4, 2, 5);
gives this message:
Too many positional parameters passed; got 3 but expected 2
in 'f' at line 3:script.p6
in main program body at line 8:script.p6
or
say f(4);
gives this message:
Not enough positional parameters passed; got 1 but expected 2
in 'f' at line 3:script.p6
in main program body at line 8:script.p6
For people who really want the Perl 5 style subroutine declaration they can use this code:
use v6;
sub f {
return @_.perl
}
say f(4, 2); # [4, 2]
say f(4); # [4]
say f(4, 2, 3); # [4, 2, 3]
but that would eliminate all the nice features of Perl 6.
What if we would like to allow the user to pass 1 or 2 parameters? For that case we can mark the second parameter as optional:
use v6;
sub f($a, $b?) {
return defined $b ?? "$a - $b" !! "$a - na";
}
say f(4, 2); # 4 - 2
say f(4); # 4 - na
The ?? !! is a the ternary operator of Perl 6 so what we see above is that if $b is defined the "$a - $b" is returned and when $b is not defined then 'na' will be returned instead of that value.
In other cases you might be tempted to replace $b with some default value like this:
use v6;
sub f($a, $b?) {
$b //= 17;
return defined $b ?? "$a - $b" !! "$a - na";
}
say f(4, 2); # 4 - 2
say f(4); # 4 - na
but this will throw an exception like this:
Cannot modify readonly value
in '&infix:<=>' at line 1
in 'f' at line 4:02.p6
in main program body at line 8:02.p6
as arguments in Perl 6 are by default read only;
There are two solutions to this, depending on what do you really want to achieve: If you only want to give a default value to $b then the best way is to add it right in the signature:
use v6;
sub f($a, $b = 17) {
return defined $b ?? "$a - $b" !! "$a - na";
}
say f(4, 2); # 4 - 2
say f(4); # 4 - 17
In this case you don't even need the question mark '?' as the presence of a default value automatically makes the parameter optional. This still keeps $b read-only within the subroutine.
The other solution is to mark the $b variable as a 'copy' of the original value. Therefore allowing the user to make changes to it:
use v6;
sub f($a, $b? is copy) {
$b //= 17;
return defined $b ?? "$a - $b" !! "$a - na";
}
say f(4, 2); # 4 - 2
say f(4); # 4 - 17
We can also restrict the parameters of a subroutine to certain data types:
use v6;
sub f($a, Int $b) {
return "$a - $b";
}
say f(4, 2);
say f(4, "foo");
The first call will succeed but the second will generate an exception:
4 - 2
Nominal type check failed for parameter '$b'; expected Int but got Str instead
in 'f' at line 4:03.p6
in main program body at line 9:03.p6
It is also possible to provide further constraints on the parameters:
use v6;
sub f($a, Int $b where { $b < 10 }) {
return "$a - $b";
}
say f(4, 2);
say f(4, 11);
4 - 2
Constraint type check failed for parameter '$b'
in 'f' at line 4:03.p6
in main program body at line 9:03.p6
There are of course a lot of other aspects for subroutine definition in Perl 6 but let's now go back to our original goal, to create the +- operator.
So how can one create a new operator in Perl 6? It is quite simple as an operator is just a subroutine with a funny way of calling it.
So here is how to create the thought operator. We create a subroutine, declaring it to be an infix operator with the name .oO
sub infix:<.oO>($name, $thought) {
say "$name thinks $thought"
}
Specifically we want to make +- an infix operator that create a Range. Not something complex. Within the angle brackets we put the new operator, the sub has a signature of two scalars and within the block we put actual code that needs to be executed.
sub infix:<+->($a, $b) { ($a-$b) .. ($a+$b) }
Then we can use
5 +- 2
that will create the range 5-2 .. 5+2 also known as 3 .. 7
I am always open to comments and criticism (just have a positive spin to it :-) So if you find any issue with the examples, please don't hesitate to let me know.
If you'd like to ask question about Perl 6, probably the best would be to sign up on the Perl 6 users list by sending an e-mail to
[email protected]
The archive of the perl6-users list is at: Perl6.org
The even better way is to join the #perl6 IRC channel on irc.freenode.net If you have not used IRC yet, the easies way is to use the web based IRC client of Freenode
Previous issues of this newsletter can be found at Perl 6 Tricks and Treats.
I was reproached by my colleague because of the lack of "no cake"-style jokes in this last grant update. So what can I do to amend the situation? Firstly, let me concede that the below blog post is almost tear-inducingly boring. Secondly, let me remind you that when accosted by boring material such as that in this post, the most important thing to have is a positive outlook on life. Thank you.
The past couple of days have been really eventful. The coming couple of days probably will be, too. It seems fitting to punctuate all this eventfulness with a status-updating blog post, something that I apparently haven't gotten around to in a few weeks.
So, what's been happening?
utf8 is an encoding, but iso-8859-1 is a charset. It's confusing me slightly, but that's OK. I can make the code work so that both are treated as "encodings" on the Perl 6 level.Buf. It all culminated in a p6l thread. Oha is writing code that uses Buf for network traffic, and has been preparing a patch for making IO::Socket::INET return Bufs instead of Strs. In trying out this patch, oha++ found that some tests will never work when Bufs are used. Those tests will probably need to be rewritten.pack and unpack in a branch. I'm making steady progress, and trying to take in documentation such as Perl 5's perldoc -f pack and perlpacktut and Erlang's ei and Tcl's binary . It's a lot to take in, and I won't aim for full functionality — look, Perl 5 has 14699 tests for pack! — but rather a reasonable subset of the directives from Perl 5.There's a lot to be said about the interplay between Perl 6's high level of abstraction and the almost reckless to-the-metal feeling of &pack/&unpack. Add to this that the "metal" in this case is Parrot, which in some regards is there to abstract away the real metal. I think the details of this interplay might well be the subject of a separate blog post. When I'm not busy finishing up the code itself. 哈哈
The hard pencils-down deadline for GSoC is on Monday. I'm pretty sure I will have tied up the remaining loose ends by then, but I also foresee a couple of fairly focused hours programming before that. Time to dig back in; see you on the other side.
In Perl, subroutine and operator names determine what happens, usually not the type of the arguments. Instead the arguments are coerced to a type on which the operation makes sense:
say uc 34; # coerces 34 to a string, and upper-cases it say 1 + "2"; # converts "2" to a number before adding
To make things more extensible, the uc function re-dispatches
to the uc method on its argument. So for the example above to
work, we need an uc function in Int. And in Array, so that
@a.uc works. And so on.
The original approach was to stuff all these methods into Any,
the base class of the object hierarchy. Which kinda worked, but also meant
that all user-defined classes ended up having some few hundreds methods to
start with. Not good.
These days, the type Cool fills this niche: most built-in types
(all that are meant to be used in that polymorphic way) inherit from Cool, so
the uc method is actually defined in class Cool,
coerces to string, and then re-dispatches to the internal logic that actually
does the upper-casing.
The name either stands for Convenient object oriented loopback, or just expresses that that most built-ins are cool with an argument of that type.
If users want to write a type that can be used like a built-in type now
just inherit from Cool, and define coercion methods to other
built-in types. If the types don't inherit from Cool, they are
more light-weight, and less magic. There's more than one way to do it.
Cool is a class (and not a role), because classes are mutable;
so if you want to inject behavior into nearly all built-in types, augmenting
Cool is an option (though usually considered evil, and should not
be done lightly).
Let's look at a week two years ago in the perl6 RT queue. Who submitted bugs?
Moritz
Moritz
Moritz
Moritz
Yaakov
Yaakov
Carl
Moritz
Moritz
Ron
Charlie
Carl
Carl
Carl
Carl
Carl
Seems I just managed to tie with Moritz++ there. Almost one ticket a day sounds about right from what I remember of my submit pace from that time. In total, there were 16 tickets in that week.
Ok, that was two years ago. What about one year ago? By this time, Rakudo had already picked up a bit of a reputation.
Patrick
Patrick
Carl
Alex
Alex
Timothy
Moritz
Gilbert
Carl
Timothy
Timothy
Eduardo
Carl
Carl
Carl
dakkar
dakkar
Hanno
Solomon
Ben
Solomon
Ben
dakkar
Aaron
Solomon
Dang, only five tickets that week. 哈哈
Still, a pretty typical week. 25 tickets all in all. We see a bit more variation in the names now. That's the new people.
Finally, let's look at last week:
Ekkehard
Timothy
Tom
cognominal
Cosimo
Sven
Chris
Carl
Carl
Carl
Moritz
Simon
Mulander
Carl
Carl
Paweł
Tadeusz
Aaron
cognominal
cognominal
Carl
Paweł
Lithos
Lithos
Carl
Carl
Tyler
Alex
Lithos
Moritz
Lithos
Brian
Moritz
Moritz
Paweł
Steve
Francesco
Lithos
Jarrod
Lithos
Francesco
Carl
Moritz
These figures help express what I feel. I'm still submitting a fair amount of tickets — all of nine last week — and still they make up an ever-smaller proportion of the growing deluge of perl6 tickets on RT.
Or, put differently, all you other people finally caught up with me. I can no longer compete with the rest of you. 哈哈
It makes me proud and excited to disappear under the waves of other people's contributions. There's been both a slow buildup over the past two years, and a sudden spike thanks to Rakudo Star. Together, they help drown out my insistent, repeating splashes on RT.
Rakudobugs are love. There's a whole lotta love out there right now. ♥ This uptake is what we've been waiting for.
As I mentioned during my lightning talk in Pisa there are a number of tech events in Europe where it would be interesting to have Perl presence. The full list of events that I am aware of can be found on the events wiki page and the Perl presence is being organized for some of them already.
If you have time it would be nice to help out the organizers or if you can attend other events, then to organize something there or if you know about more events, please add them to the wiki.
Here is what is planned so far:
Froscon in St Augustin, Germany will take place 21-22 August 2010. Organized by Renée Bäcker there is going to be a full track of Perl talks.
FrosCamp in Zurich, Switzerland will be 17-18 September 2010. Oranized by Renée Bäcker, as I can see there is going to be a Perl booth and maybe a couple of Perl related talks. The dead-line for talk submission is 2010-08-14. Just a few days from now.
OSWC in Malaga, Spain between 27-28 October 2010. The organizers, Salvador Fandiño, Diego Kuperman, JJ Merelo, Rodrigo de Oliveira are planning to have a Perl track.
Brandenburger Linux-Infotag in Potsdam, Germany on 6 Nov 2010. Organized by Renée Bäcker they are stll looking for more people to help out at the Perl booth and give Perl related talks.
T-Dose in Eindhoven, Holland will be between 6-7 November 2010. I've just started to organize the Perl presence there. Let me know if you can help out.
Since the Rakudo Star release, there has been a noticeable increase in Perl 6 questions on perlmonks - a good sign, because it means people are using it.
I've assembled this list by looking through the 100 newest nodes in Seekers of Perl Wisdom, which makes it 6% of the questions asked, or on average 1 per day (used to be around 1 per week).
Most of the questions are related to environmental issues (building/installing stuff), or beginner's questions related to syntax.
It's good to see the questions flowing in, and I hope that we'll soon see more questions where I can show off cool Perl 6 features in the answers :-).
Just saw this post. Tried to comment there, but it failed.
As I understand it, Perl 6 is supposed to have a memoizer built-in, something like sub fact is cached ($n). But that definitely isn’t implemented in Rakudo yet.
There is an interesting way to do this that does work in Rakudo:
> my $fact = gather { take 1; my $a = 1; for 1..* { $a *= $_; take $a + 0 } }; say $fact[3];
6
> say $fact[3]
6
> say $fact[4]
24
> say $fact[10]
3628800
> say $fact.munch(20).perl
(1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800, 39916800, 479001600, 6227020800, 87178291200, 1307674368000, 20922789888000, 355687428096000, 6.402373705728e+15, 1.21645100408832e+17)
That defines $fact as a lazy list of the factorials, which you can then access as, say, $fact[10] (for 10!). The list will only contain actual values for the factorials you have asked for (and any others needed to compute those).
Edited to add: Just realized there is (almost?) an easier way of doing this.
> my $fact = [\*] 1..*; say $fact[3] 24 > $fact.munch(20).perl (1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800, 39916800, 479001600, 6227020800, 87178291200, 1307674368000, 20922789888000, 355687428096000, 6.402373705728e+15, 1.21645100408832e+17, 2.43290200817664e+18)
As you can see, this is a much cleaner way of generating the list, but the indices are now off by one.
As the module database grows bigger and stronger, who wouldn’t like to see his/her work there? Here’s a bunch of notes to get You started. Please note it’s rather a piece of advice, not the standard of any sort. There is no standard about preparing modules, yet when it fullfils the below conventions it’ll have a bigger chance of working everywhere without problems. They’re based on the proto PIONEER document, but remember, there’s no standard. It’s just a convention, You can accept it or ignore it.
To create a Perl 6 module, collect the .pm files and put them all inside the lib/ directory of Your project.
“That’s all?†You’ll ask. “No metainfo files, no dist.ini, no Configure, Makefile, anything?â€. No. They’re all optional. The simple lib/ directory is enough for Your module to be easily compiled and installed by a module management tool. From now on, everything is optional. Hell, even lib/ is optional, if You just want to install some executables.
But You’ll usually need more. Tests! Everyone likes tests. You definitely want to ensure that Your module actually works. You don’t? Come on, I know You do.
So here’s another simple thing: just put all the tests in a t/ directory, and they will all be checked before the module is installed. Again, no Makefile or anything is required.
Same goes for executables. If You actually write a complete application, or just want a helper script as a bonus, put it in the bin/ directory and it will also get automagically installed. Easy peasy.
So basically You’ll end up with something like this:
And You’re done! Go and upload Your new, shiny module to Github and poke one of the proto maintainers to put it on the modules list.
But what if You actually want that Makefile? Of course You can create one, and it will be used by a module installer to ‘make’, ‘make test’ and ‘make install’ Your module. You can also include Configure.pl script (Perl 6 is preffered here), and it will be run before compilation. If You don’t really want to write Your own makefile, that problem is also solved alredy. Ufo will automagically build a perfect Makefile for You, able to compile, test, even install. If You won’t ship any Makefile with Your project, a module installer is likely to generate it with ufo anyway, so it won’t have to do everything by itself.
One last thing. Dependencies. Hell and blessing at the same time. To ensure Your dependencies will be installed before Your module is, add the deps.proto (proto.deps is also used sometimes) file in a toplevel directory, with each required module name in a new line.
So, to sum it up:
As a bonus, You can add a logotype/ directory with logo_32x32.png file. On http://modules.perl6.org/ You can see how it will appear in the modules database.
It’s nice to see how the existing modules look like. Math::Model and Yapsi are good examples.
Remember, these aren’t the rules, and there is no standard. It’s just nice to have it simple and not to overcomplicate too much.
That’s all for today. Happy hacking! Fun, fame, and ladies’ hearts await!
(Forgive me starting from the nerdy stuff. Give me some time to get comfortable and I may write about something interesting one day)
Rakudo Star, being “useful and usableâ€, brought not only hapiness and amazement, but also suprisingly many votes of disapproval, even from inside the Perl community. It’s slow, it’s not complete, it’s this and that, blah blah blah. I don’t know what these people were actually expecting, but I’m happy about what the Rakudo team has released. And here’s why.
I’m not a proffesional programmer of any sort (hell, I won’t even go so far to call myself a programmer at all). It’s my passion, not my job, and a die-hard production-ready Perl 6 is not what’s obligatory for me. As Perl 6, I’m also optimized for fun, so as long as I can do funny stuff with it, and I can, I’m glad, happy, dancing around and juggling ananases.
So I’m playing around. Writing modules, like File::Find, toying with module management (see neutro) and messing with other people’s repos. I’m having fun, gaining experience and I believe I’m doing something others will appreciate. What else do I need?
At YAPC::EU 2010 we had a long discussion about Perl 6, Rakudo and related matters. Here are some (very incomplete) notes of the ongoing discussions and results.
Patrick Michaud, Jonathan Worthington, Carl Mäsak, Moritz Lenz, Gabor Szabo, and a fluctuation of other Perl 6 hackers.
What can we do to improve Rakudo's performance?
See http://news.perlfoundation.org/2010/07/hague-grant-application-meta-m.html. Will probably bring the biggest speed improvement of all options we have under our control
Most Rakudo built-ins are written for correctness first, and without a good feeling for what constructs are fast and what aren't. A thorough review (and preferably profiling) could bring decent speed improvements, as the case of int ranges showed.
Parrot's GC is... suboptimal. To be gentle.
We will try to convince people that Tyler Curtis' optimization framework for
PAST and POST should be shipped with parrot (probably compile PIRs in ext/,
just like NQP-rx does it now). Using that, we can do constant folding
Number parsing needs to be moved to compile time.
Brought up by Gabor
We do much volunteer work, but when we get funding, we can devote more time to hacking
We'd like to get together a few times (2? 3? 4?) a year, in real life.
Funding and organization would be very welcome
It would be nice to have a way to have funding available much more quickly than through the usual grant process, which tends to be longish.
Good: It worked. It did what we wanted it to.
Bad:
It lacked a module installer (It shipped proto, but didn't install it).
Compilation takes too much memory. pmichaud will try a hack to split the setting, which would solve that problem.
There was some discussion about the roles + outer scopes bugs, which was way over my head. It seems to be related to the fact that parrot has two outer chains for nested blocks: one at compile time, one at runtime. Since role methods are flattened into classes, there compile time outer block is actually different than where it runs, and that screws up ... forget it, somebody else must describe it.
Lack of modules - doesn't seem to bee a big problem
Lack of features: not a big problem.
Biggest complaints: missing perl6doc. Missing non-blocking IO, binary file support.
Prefix paths with spaces are not supported :(
jnthn: "I actually tried to write a C program that binary patches the perl6 executable to allow spaces in path names. It almost worked."
We will try to advocate compilation to PBC, not PIR - once that's supported.
Proto needs to be end-of-life'd.
It confuses people that there are two different project lists, and the lists diverge.
We would like to decentralize the module list somehow. Still open how.
People don't release Perl 6 modules, because there's no need so far, and it's tedious to add the version name in each .pm/.pm6 file. We might need to come up with a clever idea for that.
Additionally to the parrot backend, we want to run Perl 6 code on other virtual machines.
jnthn will work on a .NET/CLR port. He wants to prototype the new low-level class composition code in .NET anyway, which will provide the basic foundations for running NQP.
pmichaud wants to explore javascript on V8 as a possible backend. "I managed PIR, I'll certainly manage javascript" :-)
Huge time sink, but still worth doing it
Apache runtime library might be worth looking into
risks: stalled refactors are dangerous (see: PHP 6, cardinal (the ruby-on-parrot compiler))
We want to avoid fragmentation into many subprojects
We want to increase the number of possible contributors to rakudo by enabling non-parrot people to contribute.
Code for different backends will be maintained as directories in Rakudo and NQP, not as branches.
pir:: things will be hidden behind an nqp:: abstraction layer
Moritz wants to continue with the "weekly" challenges, but runs out of ideas. Add ideas to http://svn.pugscode.org/pugs/misc/helpnow/README.
We will try to apply patches faster, thus encouraging people who already did the first step.
in p5 pod for now, so that people can contribute easily
masak and szabgab expressed interest in working on pod6 tools