Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • World
  • Users
  • Groups
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

NodeBB

  1. Home
  2. uncategorized
  3. I'm now writing a JSON parser in Koka.

I'm now writing a JSON parser in Koka.

Scheduled Pinned Locked Moved uncategorized
31 Posts 12 Posters 4 Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

    Well, guess what
    I need to write a module for unit testing, too
    ​​

    volpeon@icy.wyvern.ripV This user is from outside of this forum
    volpeon@icy.wyvern.ripV This user is from outside of this forum
    volpeon@icy.wyvern.rip
    wrote last edited by
    #22

    Well, guess what
    The community stdlib has one and it's good, so they spared me the effort
    ​​

    volpeon@icy.wyvern.ripV 1 Reply Last reply
    0
    • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

      Well, guess what
      The community stdlib has one and it's good, so they spared me the effort
      ​​

      volpeon@icy.wyvern.ripV This user is from outside of this forum
      volpeon@icy.wyvern.ripV This user is from outside of this forum
      volpeon@icy.wyvern.rip
      wrote last edited by volpeon@icy.wyvern.rip
      #23

      I was about to add unit tests for the IRI parser and found a nice JSON file with lots of test cases. I have JSON parsing, so that's good. What I don't have is a nice way to decode the generic json type to a specific type presentation, like with Aeson's FromJSON. I could still get all the test cases from the JSON file without it, but the rest of the application needs to handle JSON as well and then a decoder abstraction would be really good to have. So writing a decoder module is next ​:brdFlat:​

      edit: Never mind again. I keep missing stuff because I keep looking for Haskell-like solutions, but Koka is simply different.

      volpeon@icy.wyvern.ripV 1 Reply Last reply
      0
      • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

        I was about to add unit tests for the IRI parser and found a nice JSON file with lots of test cases. I have JSON parsing, so that's good. What I don't have is a nice way to decode the generic json type to a specific type presentation, like with Aeson's FromJSON. I could still get all the test cases from the JSON file without it, but the rest of the application needs to handle JSON as well and then a decoder abstraction would be really good to have. So writing a decoder module is next ​:brdFlat:​

        edit: Never mind again. I keep missing stuff because I keep looking for Haskell-like solutions, but Koka is simply different.

        volpeon@icy.wyvern.ripV This user is from outside of this forum
        volpeon@icy.wyvern.ripV This user is from outside of this forum
        volpeon@icy.wyvern.rip
        wrote last edited by volpeon@icy.wyvern.rip
        #24

        The application crashes when the JSON is too large. I bet it's because Koka's parser combinator library makes every single step backtrackable.

        volpeon@icy.wyvern.ripV 1 Reply Last reply
        0
        • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

          The application crashes when the JSON is too large. I bet it's because Koka's parser combinator library makes every single step backtrackable.

          volpeon@icy.wyvern.ripV This user is from outside of this forum
          volpeon@icy.wyvern.ripV This user is from outside of this forum
          volpeon@icy.wyvern.rip
          wrote last edited by
          #25

          It's actually not too difficult to avoid backtracking if you make the subparsers smarter. I just didn't do it because it also makes them less readable, but oh well.
          So right now I parse booleans like this, for instance:
          string("true") || string("false"), which means it tries the first parser, and on failure the second one (and therefore the consumed input must get "un-consumed").
          The smarter solution is to pick the right branch directly so there is no "un-consuming" necessary. Like this:

          val c = one-of("tf")
          if c == 't' then
            string("rue")
          else
            string("alse")

          volpeon@icy.wyvern.ripV 1 Reply Last reply
          0
          • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

            It's actually not too difficult to avoid backtracking if you make the subparsers smarter. I just didn't do it because it also makes them less readable, but oh well.
            So right now I parse booleans like this, for instance:
            string("true") || string("false"), which means it tries the first parser, and on failure the second one (and therefore the consumed input must get "un-consumed").
            The smarter solution is to pick the right branch directly so there is no "un-consuming" necessary. Like this:

            val c = one-of("tf")
            if c == 't' then
              string("rue")
            else
              string("alse")

            volpeon@icy.wyvern.ripV This user is from outside of this forum
            volpeon@icy.wyvern.ripV This user is from outside of this forum
            volpeon@icy.wyvern.rip
            wrote last edited by
            #26

            I removed all backtracking and it can handle larger JSON files now, but I still need to remove a very large chunk of the tests file to avoid the crash. ​​

            volpeon@icy.wyvern.ripV legion495@mk.absturztau.beL 2 Replies Last reply
            0
            • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

              I removed all backtracking and it can handle larger JSON files now, but I still need to remove a very large chunk of the tests file to avoid the crash. ​​

              volpeon@icy.wyvern.ripV This user is from outside of this forum
              volpeon@icy.wyvern.ripV This user is from outside of this forum
              volpeon@icy.wyvern.rip
              wrote last edited by
              #27

              Looks like it's a known issue with ctl effects (which parsers use): github.com/koka-lang/koka/issues/279
              There's one more idea I want to try, but otherwise I'll just accept it for now

              volpeon@icy.wyvern.ripV 1 Reply Last reply
              0
              • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                I removed all backtracking and it can handle larger JSON files now, but I still need to remove a very large chunk of the tests file to avoid the crash. ​​

                legion495@mk.absturztau.beL This user is from outside of this forum
                legion495@mk.absturztau.beL This user is from outside of this forum
                legion495@mk.absturztau.be
                wrote last edited by
                #28

                @volpeon@icy.wyvern.rip How large exactly? ​​

                1 Reply Last reply
                0
                • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                  Looks like it's a known issue with ctl effects (which parsers use): github.com/koka-lang/koka/issues/279
                  There's one more idea I want to try, but otherwise I'll just accept it for now

                  volpeon@icy.wyvern.ripV This user is from outside of this forum
                  volpeon@icy.wyvern.ripV This user is from outside of this forum
                  volpeon@icy.wyvern.rip
                  wrote last edited by
                  #29

                  IRI parser works ​​
                  Only the IRI resolution algorithm left, and then I can finally get to JSON-LD

                  johann150@genau.qwertqwefsday.euJ volpeon@icy.wyvern.ripV 2 Replies Last reply
                  0
                  • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                    IRI parser works ​​
                    Only the IRI resolution algorithm left, and then I can finally get to JSON-LD

                    johann150@genau.qwertqwefsday.euJ This user is from outside of this forum
                    johann150@genau.qwertqwefsday.euJ This user is from outside of this forum
                    johann150@genau.qwertqwefsday.eu
                    wrote last edited by
                    #30

                    @volpeon@icy.wyvern.rip oh no, gonna loose volpeon to JSON-LD again

                    will it be like the last time you tried that?(i.e. we don't hear from you at all for a week or so)
                    ​​

                    1 Reply Last reply
                    0
                    • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                      IRI parser works ​​
                      Only the IRI resolution algorithm left, and then I can finally get to JSON-LD

                      volpeon@icy.wyvern.ripV This user is from outside of this forum
                      volpeon@icy.wyvern.ripV This user is from outside of this forum
                      volpeon@icy.wyvern.rip
                      wrote last edited by
                      #31

                      I revisited the JSON parser and wrote my own parsing library. It's not too different from the one in the stdlib, but it handles some cases more efficiently and that makes a huge difference.
                      The old parser caused a stack overflow with a 144kb JSON file. The new parser can handle a 1mb JSON test file, but still causes a stack overflow with a 5mb file.

                      1 Reply Last reply
                      0
                      Reply
                      • Reply as topic
                      Log in to reply
                      • Oldest to Newest
                      • Newest to Oldest
                      • Most Votes


                      • Login

                      • Login or register to search.
                      Powered by NodeBB Contributors
                      • First post
                        Last post
                      0
                      • Categories
                      • Recent
                      • Tags
                      • Popular
                      • World
                      • Users
                      • Groups