Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • World
  • Users
  • Groups
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

NodeBB

  1. Home
  2. uncategorized
  3. I'm now writing a JSON parser in Koka.

I'm now writing a JSON parser in Koka.

Scheduled Pinned Locked Moved uncategorized
31 Posts 12 Posters 4 Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

    Since Koka has no network stdlib yet, I'll have to implement URI and IRI handling first ​​

    volpeon@icy.wyvern.ripV This user is from outside of this forum
    volpeon@icy.wyvern.ripV This user is from outside of this forum
    volpeon@icy.wyvern.rip
    wrote last edited by volpeon@icy.wyvern.rip
    #19

    Positive side effect compared to Haskell: I can unify the URI and IRI code because if there aren't any standards, I am the one making them ​:brdCool:​ In Haskell, they were in separate packages and it was kinda clunky

    1 Reply Last reply
    0
    • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

      Since Koka has no network stdlib yet, I'll have to implement URI and IRI handling first ​​

      volpeon@icy.wyvern.ripV This user is from outside of this forum
      volpeon@icy.wyvern.ripV This user is from outside of this forum
      volpeon@icy.wyvern.rip
      wrote last edited by
      #20

      Well, guess what
      I need to write a module for unit testing, too
      ​​

      legion495@mk.absturztau.beL volpeon@icy.wyvern.ripV 2 Replies Last reply
      0
      • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

        Well, guess what
        I need to write a module for unit testing, too
        ​​

        legion495@mk.absturztau.beL This user is from outside of this forum
        legion495@mk.absturztau.beL This user is from outside of this forum
        legion495@mk.absturztau.be
        wrote last edited by
        #21

        @volpeon@icy.wyvern.rip An absolut unit by birb

        1 Reply Last reply
        0
        • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

          Well, guess what
          I need to write a module for unit testing, too
          ​​

          volpeon@icy.wyvern.ripV This user is from outside of this forum
          volpeon@icy.wyvern.ripV This user is from outside of this forum
          volpeon@icy.wyvern.rip
          wrote last edited by
          #22

          Well, guess what
          The community stdlib has one and it's good, so they spared me the effort
          ​​

          volpeon@icy.wyvern.ripV 1 Reply Last reply
          0
          • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

            Well, guess what
            The community stdlib has one and it's good, so they spared me the effort
            ​​

            volpeon@icy.wyvern.ripV This user is from outside of this forum
            volpeon@icy.wyvern.ripV This user is from outside of this forum
            volpeon@icy.wyvern.rip
            wrote last edited by volpeon@icy.wyvern.rip
            #23

            I was about to add unit tests for the IRI parser and found a nice JSON file with lots of test cases. I have JSON parsing, so that's good. What I don't have is a nice way to decode the generic json type to a specific type presentation, like with Aeson's FromJSON. I could still get all the test cases from the JSON file without it, but the rest of the application needs to handle JSON as well and then a decoder abstraction would be really good to have. So writing a decoder module is next ​:brdFlat:​

            edit: Never mind again. I keep missing stuff because I keep looking for Haskell-like solutions, but Koka is simply different.

            volpeon@icy.wyvern.ripV 1 Reply Last reply
            0
            • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

              I was about to add unit tests for the IRI parser and found a nice JSON file with lots of test cases. I have JSON parsing, so that's good. What I don't have is a nice way to decode the generic json type to a specific type presentation, like with Aeson's FromJSON. I could still get all the test cases from the JSON file without it, but the rest of the application needs to handle JSON as well and then a decoder abstraction would be really good to have. So writing a decoder module is next ​:brdFlat:​

              edit: Never mind again. I keep missing stuff because I keep looking for Haskell-like solutions, but Koka is simply different.

              volpeon@icy.wyvern.ripV This user is from outside of this forum
              volpeon@icy.wyvern.ripV This user is from outside of this forum
              volpeon@icy.wyvern.rip
              wrote last edited by volpeon@icy.wyvern.rip
              #24

              The application crashes when the JSON is too large. I bet it's because Koka's parser combinator library makes every single step backtrackable.

              volpeon@icy.wyvern.ripV 1 Reply Last reply
              0
              • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                The application crashes when the JSON is too large. I bet it's because Koka's parser combinator library makes every single step backtrackable.

                volpeon@icy.wyvern.ripV This user is from outside of this forum
                volpeon@icy.wyvern.ripV This user is from outside of this forum
                volpeon@icy.wyvern.rip
                wrote last edited by
                #25

                It's actually not too difficult to avoid backtracking if you make the subparsers smarter. I just didn't do it because it also makes them less readable, but oh well.
                So right now I parse booleans like this, for instance:
                string("true") || string("false"), which means it tries the first parser, and on failure the second one (and therefore the consumed input must get "un-consumed").
                The smarter solution is to pick the right branch directly so there is no "un-consuming" necessary. Like this:

                val c = one-of("tf")
                if c == 't' then
                  string("rue")
                else
                  string("alse")

                volpeon@icy.wyvern.ripV 1 Reply Last reply
                0
                • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                  It's actually not too difficult to avoid backtracking if you make the subparsers smarter. I just didn't do it because it also makes them less readable, but oh well.
                  So right now I parse booleans like this, for instance:
                  string("true") || string("false"), which means it tries the first parser, and on failure the second one (and therefore the consumed input must get "un-consumed").
                  The smarter solution is to pick the right branch directly so there is no "un-consuming" necessary. Like this:

                  val c = one-of("tf")
                  if c == 't' then
                    string("rue")
                  else
                    string("alse")

                  volpeon@icy.wyvern.ripV This user is from outside of this forum
                  volpeon@icy.wyvern.ripV This user is from outside of this forum
                  volpeon@icy.wyvern.rip
                  wrote last edited by
                  #26

                  I removed all backtracking and it can handle larger JSON files now, but I still need to remove a very large chunk of the tests file to avoid the crash. ​​

                  volpeon@icy.wyvern.ripV legion495@mk.absturztau.beL 2 Replies Last reply
                  0
                  • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                    I removed all backtracking and it can handle larger JSON files now, but I still need to remove a very large chunk of the tests file to avoid the crash. ​​

                    volpeon@icy.wyvern.ripV This user is from outside of this forum
                    volpeon@icy.wyvern.ripV This user is from outside of this forum
                    volpeon@icy.wyvern.rip
                    wrote last edited by
                    #27

                    Looks like it's a known issue with ctl effects (which parsers use): github.com/koka-lang/koka/issues/279
                    There's one more idea I want to try, but otherwise I'll just accept it for now

                    volpeon@icy.wyvern.ripV 1 Reply Last reply
                    0
                    • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                      I removed all backtracking and it can handle larger JSON files now, but I still need to remove a very large chunk of the tests file to avoid the crash. ​​

                      legion495@mk.absturztau.beL This user is from outside of this forum
                      legion495@mk.absturztau.beL This user is from outside of this forum
                      legion495@mk.absturztau.be
                      wrote last edited by
                      #28

                      @volpeon@icy.wyvern.rip How large exactly? ​​

                      1 Reply Last reply
                      0
                      • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                        Looks like it's a known issue with ctl effects (which parsers use): github.com/koka-lang/koka/issues/279
                        There's one more idea I want to try, but otherwise I'll just accept it for now

                        volpeon@icy.wyvern.ripV This user is from outside of this forum
                        volpeon@icy.wyvern.ripV This user is from outside of this forum
                        volpeon@icy.wyvern.rip
                        wrote last edited by
                        #29

                        IRI parser works ​​
                        Only the IRI resolution algorithm left, and then I can finally get to JSON-LD

                        johann150@genau.qwertqwefsday.euJ volpeon@icy.wyvern.ripV 2 Replies Last reply
                        0
                        • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                          IRI parser works ​​
                          Only the IRI resolution algorithm left, and then I can finally get to JSON-LD

                          johann150@genau.qwertqwefsday.euJ This user is from outside of this forum
                          johann150@genau.qwertqwefsday.euJ This user is from outside of this forum
                          johann150@genau.qwertqwefsday.eu
                          wrote last edited by
                          #30

                          @volpeon@icy.wyvern.rip oh no, gonna loose volpeon to JSON-LD again

                          will it be like the last time you tried that?(i.e. we don't hear from you at all for a week or so)
                          ​​

                          1 Reply Last reply
                          0
                          • volpeon@icy.wyvern.ripV volpeon@icy.wyvern.rip

                            IRI parser works ​​
                            Only the IRI resolution algorithm left, and then I can finally get to JSON-LD

                            volpeon@icy.wyvern.ripV This user is from outside of this forum
                            volpeon@icy.wyvern.ripV This user is from outside of this forum
                            volpeon@icy.wyvern.rip
                            wrote last edited by
                            #31

                            I revisited the JSON parser and wrote my own parsing library. It's not too different from the one in the stdlib, but it handles some cases more efficiently and that makes a huge difference.
                            The old parser caused a stack overflow with a 144kb JSON file. The new parser can handle a 1mb JSON test file, but still causes a stack overflow with a 5mb file.

                            1 Reply Last reply
                            0
                            Reply
                            • Reply as topic
                            Log in to reply
                            • Oldest to Newest
                            • Newest to Oldest
                            • Most Votes


                            • Login

                            • Login or register to search.
                            Powered by NodeBB Contributors
                            • First post
                              Last post
                            0
                            • Categories
                            • Recent
                            • Tags
                            • Popular
                            • World
                            • Users
                            • Groups