I'm now writing a JSON parser in Koka.
-
@volpeon going down the next rabbit hole?
@stefan The grind never ends
-
Since Koka has no network stdlib yet, I'll have to implement URI and IRI handling first
Positive side effect compared to Haskell: I can unify the URI and IRI code because if there aren't any standards, I am the one making them :brdCool: In Haskell, they were in separate packages and it was kinda clunky
-
Since Koka has no network stdlib yet, I'll have to implement URI and IRI handling first
Well, guess what
I need to write a module for unit testing, too 
-
Well, guess what
I need to write a module for unit testing, too @volpeon@icy.wyvern.rip An absolut unit by birb
-
Well, guess what
I need to write a module for unit testing, too Well, guess what
The community stdlib has one and it's good, so they spared me the effort -
Well, guess what
The community stdlib has one and it's good, so they spared me the effort I was about to add unit tests for the IRI parser and found a nice JSON file with lots of test cases. I have JSON parsing, so that's good. What I don't have is a nice way to decode the generic json type to a specific type presentation, like with Aeson's FromJSON. I could still get all the test cases from the JSON file without it, but the rest of the application needs to handle JSON as well and then a decoder abstraction would be really good to have. So writing a decoder module is next :brdFlat:
edit: Never mind again. I keep missing stuff because I keep looking for Haskell-like solutions, but Koka is simply different. -
I was about to add unit tests for the IRI parser and found a nice JSON file with lots of test cases. I have JSON parsing, so that's good. What I don't have is a nice way to decode the generic json type to a specific type presentation, like with Aeson's FromJSON. I could still get all the test cases from the JSON file without it, but the rest of the application needs to handle JSON as well and then a decoder abstraction would be really good to have. So writing a decoder module is next :brdFlat:
edit: Never mind again. I keep missing stuff because I keep looking for Haskell-like solutions, but Koka is simply different.The application crashes when the JSON is too large. I bet it's because Koka's parser combinator library makes every single step backtrackable.
-
The application crashes when the JSON is too large. I bet it's because Koka's parser combinator library makes every single step backtrackable.
It's actually not too difficult to avoid backtracking if you make the subparsers smarter. I just didn't do it because it also makes them less readable, but oh well.
So right now I parse booleans like this, for instance:string("true") || string("false"), which means it tries the first parser, and on failure the second one (and therefore the consumed input must get "un-consumed").
The smarter solution is to pick the right branch directly so there is no "un-consuming" necessary. Like this:val c = one-of("tf") if c == 't' then string("rue") else string("alse") -
It's actually not too difficult to avoid backtracking if you make the subparsers smarter. I just didn't do it because it also makes them less readable, but oh well.
So right now I parse booleans like this, for instance:string("true") || string("false"), which means it tries the first parser, and on failure the second one (and therefore the consumed input must get "un-consumed").
The smarter solution is to pick the right branch directly so there is no "un-consuming" necessary. Like this:val c = one-of("tf") if c == 't' then string("rue") else string("alse")I removed all backtracking and it can handle larger JSON files now, but I still need to remove a very large chunk of the tests file to avoid the crash.
-
I removed all backtracking and it can handle larger JSON files now, but I still need to remove a very large chunk of the tests file to avoid the crash.
Looks like it's a known issue with ctl effects (which parsers use): github.com/koka-lang/koka/issues/279
There's one more idea I want to try, but otherwise I'll just accept it for now -
I removed all backtracking and it can handle larger JSON files now, but I still need to remove a very large chunk of the tests file to avoid the crash.
@volpeon@icy.wyvern.rip How large exactly?
-
Looks like it's a known issue with ctl effects (which parsers use): github.com/koka-lang/koka/issues/279
There's one more idea I want to try, but otherwise I'll just accept it for nowIRI parser works
Only the IRI resolution algorithm left, and then I can finally get to JSON-LD -
IRI parser works
Only the IRI resolution algorithm left, and then I can finally get to JSON-LD@volpeon@icy.wyvern.rip oh no, gonna loose volpeon to JSON-LD again
will it be like the last time you tried that?(i.e. we don't hear from you at all for a week or so)
-
IRI parser works
Only the IRI resolution algorithm left, and then I can finally get to JSON-LDI revisited the JSON parser and wrote my own parsing library. It's not too different from the one in the stdlib, but it handles some cases more efficiently and that makes a huge difference.
The old parser caused a stack overflow with a 144kb JSON file. The new parser can handle a 1mb JSON test file, but still causes a stack overflow with a 5mb file.
