Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • World
  • Users
  • Groups
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

NodeBB

  1. Home
  2. uncategorized
  3. I'm now writing a JSON parser in Koka.

I'm now writing a JSON parser in Koka.

Scheduled Pinned Locked Moved uncategorized
31 Posts 12 Posters 4 Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

    Well, guess what
    I need to write a module for unit testing, too
    ​​

    legion495@mk.absturztau.beL This user is from outside of this forum
    legion495@mk.absturztau.beL This user is from outside of this forum
    legion495@mk.absturztau.be
    wrote last edited by
    #21

    @volpeon@icy.wyvern.rip An absolut unit by birb

    1 Reply Last reply
    0
    • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

      Well, guess what
      I need to write a module for unit testing, too
      ​​

      volpeon@icy.wyvern.ripV This user is from outside of this forum
      volpeon@icy.wyvern.ripV This user is from outside of this forum
      volpeon@icy.wyvern.rip
      wrote last edited by
      #22

      Well, guess what
      The community stdlib has one and it's good, so they spared me the effort
      ​​

      volpeon@icy.wyvern.ripV 1 Reply Last reply
      0
      • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

        Well, guess what
        The community stdlib has one and it's good, so they spared me the effort
        ​​

        volpeon@icy.wyvern.ripV This user is from outside of this forum
        volpeon@icy.wyvern.ripV This user is from outside of this forum
        volpeon@icy.wyvern.rip
        wrote last edited by volpeon@icy.wyvern.rip
        #23

        I was about to add unit tests for the IRI parser and found a nice JSON file with lots of test cases. I have JSON parsing, so that's good. What I don't have is a nice way to decode the generic json type to a specific type presentation, like with Aeson's FromJSON. I could still get all the test cases from the JSON file without it, but the rest of the application needs to handle JSON as well and then a decoder abstraction would be really good to have. So writing a decoder module is next ​:brdFlat:​

        edit: Never mind again. I keep missing stuff because I keep looking for Haskell-like solutions, but Koka is simply different.

        volpeon@icy.wyvern.ripV 1 Reply Last reply
        0
        • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

          I was about to add unit tests for the IRI parser and found a nice JSON file with lots of test cases. I have JSON parsing, so that's good. What I don't have is a nice way to decode the generic json type to a specific type presentation, like with Aeson's FromJSON. I could still get all the test cases from the JSON file without it, but the rest of the application needs to handle JSON as well and then a decoder abstraction would be really good to have. So writing a decoder module is next ​:brdFlat:​

          edit: Never mind again. I keep missing stuff because I keep looking for Haskell-like solutions, but Koka is simply different.

          volpeon@icy.wyvern.ripV This user is from outside of this forum
          volpeon@icy.wyvern.ripV This user is from outside of this forum
          volpeon@icy.wyvern.rip
          wrote last edited by volpeon@icy.wyvern.rip
          #24

          The application crashes when the JSON is too large. I bet it's because Koka's parser combinator library makes every single step backtrackable.

          volpeon@icy.wyvern.ripV 1 Reply Last reply
          0
          • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

            The application crashes when the JSON is too large. I bet it's because Koka's parser combinator library makes every single step backtrackable.

            volpeon@icy.wyvern.ripV This user is from outside of this forum
            volpeon@icy.wyvern.ripV This user is from outside of this forum
            volpeon@icy.wyvern.rip
            wrote last edited by
            #25

            It's actually not too difficult to avoid backtracking if you make the subparsers smarter. I just didn't do it because it also makes them less readable, but oh well.
            So right now I parse booleans like this, for instance:
            string("true") || string("false"), which means it tries the first parser, and on failure the second one (and therefore the consumed input must get "un-consumed").
            The smarter solution is to pick the right branch directly so there is no "un-consuming" necessary. Like this:

            val c = one-of("tf")
            if c == 't' then
              string("rue")
            else
              string("alse")

            volpeon@icy.wyvern.ripV 1 Reply Last reply
            0
            • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

              It's actually not too difficult to avoid backtracking if you make the subparsers smarter. I just didn't do it because it also makes them less readable, but oh well.
              So right now I parse booleans like this, for instance:
              string("true") || string("false"), which means it tries the first parser, and on failure the second one (and therefore the consumed input must get "un-consumed").
              The smarter solution is to pick the right branch directly so there is no "un-consuming" necessary. Like this:

              val c = one-of("tf")
              if c == 't' then
                string("rue")
              else
                string("alse")

              volpeon@icy.wyvern.ripV This user is from outside of this forum
              volpeon@icy.wyvern.ripV This user is from outside of this forum
              volpeon@icy.wyvern.rip
              wrote last edited by
              #26

              I removed all backtracking and it can handle larger JSON files now, but I still need to remove a very large chunk of the tests file to avoid the crash. ​​

              volpeon@icy.wyvern.ripV legion495@mk.absturztau.beL 2 Replies Last reply
              0
              • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                I removed all backtracking and it can handle larger JSON files now, but I still need to remove a very large chunk of the tests file to avoid the crash. ​​

                volpeon@icy.wyvern.ripV This user is from outside of this forum
                volpeon@icy.wyvern.ripV This user is from outside of this forum
                volpeon@icy.wyvern.rip
                wrote last edited by
                #27

                Looks like it's a known issue with ctl effects (which parsers use): github.com/koka-lang/koka/issues/279
                There's one more idea I want to try, but otherwise I'll just accept it for now

                volpeon@icy.wyvern.ripV 1 Reply Last reply
                0
                • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                  I removed all backtracking and it can handle larger JSON files now, but I still need to remove a very large chunk of the tests file to avoid the crash. ​​

                  legion495@mk.absturztau.beL This user is from outside of this forum
                  legion495@mk.absturztau.beL This user is from outside of this forum
                  legion495@mk.absturztau.be
                  wrote last edited by
                  #28

                  @volpeon@icy.wyvern.rip How large exactly? ​​

                  1 Reply Last reply
                  0
                  • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                    Looks like it's a known issue with ctl effects (which parsers use): github.com/koka-lang/koka/issues/279
                    There's one more idea I want to try, but otherwise I'll just accept it for now

                    volpeon@icy.wyvern.ripV This user is from outside of this forum
                    volpeon@icy.wyvern.ripV This user is from outside of this forum
                    volpeon@icy.wyvern.rip
                    wrote last edited by
                    #29

                    IRI parser works ​​
                    Only the IRI resolution algorithm left, and then I can finally get to JSON-LD

                    johann150@genau.qwertqwefsday.euJ volpeon@icy.wyvern.ripV 2 Replies Last reply
                    0
                    • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                      IRI parser works ​​
                      Only the IRI resolution algorithm left, and then I can finally get to JSON-LD

                      johann150@genau.qwertqwefsday.euJ This user is from outside of this forum
                      johann150@genau.qwertqwefsday.euJ This user is from outside of this forum
                      johann150@genau.qwertqwefsday.eu
                      wrote last edited by
                      #30

                      @volpeon@icy.wyvern.rip oh no, gonna loose volpeon to JSON-LD again

                      will it be like the last time you tried that?(i.e. we don't hear from you at all for a week or so)
                      ​​

                      1 Reply Last reply
                      0
                      • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                        IRI parser works ​​
                        Only the IRI resolution algorithm left, and then I can finally get to JSON-LD

                        volpeon@icy.wyvern.ripV This user is from outside of this forum
                        volpeon@icy.wyvern.ripV This user is from outside of this forum
                        volpeon@icy.wyvern.rip
                        wrote last edited by
                        #31

                        I revisited the JSON parser and wrote my own parsing library. It's not too different from the one in the stdlib, but it handles some cases more efficiently and that makes a huge difference.
                        The old parser caused a stack overflow with a 144kb JSON file. The new parser can handle a 1mb JSON test file, but still causes a stack overflow with a 5mb file.

                        1 Reply Last reply
                        0
                        Reply
                        • Reply as topic
                        Log in to reply
                        • Oldest to Newest
                        • Newest to Oldest
                        • Most Votes


                        • Login

                        • Login or register to search.
                        Powered by NodeBB Contributors
                        • First post
                          Last post
                        0
                        • Categories
                        • Recent
                        • Tags
                        • Popular
                        • World
                        • Users
                        • Groups