This week we discuss all the work that is taking place on the documentation and new types.
Welcome to Dev Log 141. I'm going to warn you up front, this log is a bit dull :) Because I've spent the week working like mad on the documentation, which despite being vital, doesn't make for very interesting reading in a dev log.
Documentation is important for two reasons. First, and primarily, it's the thing a lot of you turn to when looking for answers. I know for a fact that a number of you go to the source code first and will manually look at a function to see what it does. But to do that, you need a good understanding of the layout of the API, and learning that takes time. So the majority of you, quite rightly, go to the API docs.
I've been working tirelessly this year on improving the Phaser 3 docs. From running the Doc Jam, to the days spent refactoring the events, to enhancing them with every release. Not a single new feature has been added to the API that isn't completely documented and it's been this way for many months now. That doesn't mean there aren't still areas that need completing - and the overall general way you browse the docs still leaves a huge amount to be desired.
The cause of this is two-fold. First, the actual docs themselves (i.e. the JSDoc blocks within the source code) need improving and expanding. There are still some areas left that don't have documentation at all, although these get less and less every month. Secondly, the HTML you browse when you visit the docs site is the direct output of a JSDoc template using the Cerulean theme. And honestly, it's far from great.
This is no fault of the template. After all, just like Phaser, it can't be everything to everyone. The overall layout, though, is just wrong. Making your own JSDoc templates is an arcane art. They're a bunch of static files that make use of a custom templating system, with HTML injected into JS injected into html. It's quite something to behold, let alone work with. The bigger issue, though, is that it takes a significant amount of time to actually generate the documentation. It will build the docs for the entire API, every single time, which generates hundreds and hundreds of files. 65MB worth each time in fact, taking several minutes. Which is less than ideal if you're making a quick tweak and want to test it to see if it looks right. Again, this isn't the fault of JSDoc. It is designed to generate static doc files from your source, that is its whole purpose. But it's not what Phaser needs.
Alert long-time readers may remember that back in Dev Log 100 I wrote about the work I was doing on the docs. At the time I was mostly talking about the app I built called Percy, which helped me generate all the docblocks the code required. However, I also alluded to the fact that I wanted to release the Phaser docs as a SQLite file and build a front-end around that. I spent a couple of days work on this at the time, but it never got any further because I was too wrapped up in actually getting Phaser itself working and the docs were written. Fast forward to this week, though, and that is where my work has been focused.
Types Ahoy
The first stage was sorting out all of the types. JSDoc offers a feature called a @typedef which allows you to create a new custom type that you then reference in the JSDocs in your code. I had been using these already, but they weren't properly namespaced, the location of the defs was inconsistent and the use of them was patchy. So the week began by addressing this.
For all areas that use them I created typedefs folders and then created a single file per type in there. For example, the Animation namespace has 7 unique typedefs at the moment, so I moved the defs out of the source code and into that folder:
Each of those JS files only contains a JSDoc typedef block. There is no code in them, so they don't add anything to the library size. Here is the typedef file for the Animation object:
As you can see it's a description of an Animation object with all of the properties listed and described. Having it like this means I can reference this type in the rest of the docs, such as in the AnimationManager.create method:
In the code above you can see that the parameter needs to be an Animation configuration object. Previously, before typedefs, lots of methods this like would have just said 'object' as the type. With typedefs you now know the exact shape of the object it expects. This should prove especially useful for TypeScript users too.
Of course, creating all of these types was a lot of work. And it was that mind-numbing labor-intensive type of work, too. Absolutely zero fun, zero creativity, no challenge. Just hour after hour, day after day, of cutting and pasting. The things I do for you guys :) Ultimately, it was worth it. It instantly made the JSDocs better. Even using the current lacklustre template it improves it hugely, because types are organized into a single place per namespace, and you can click them when viewing a method that uses one as a parameter and get taken right to the definition for it.
So, time well spent, despite the monotony of it. I find this to be a recurring trend when it comes to Phaser, or I guess any large scale open source project. There's the fun creative side of it, like adding cool new features such as improving something in the renderer. There's the challenging 'triumphant' side of it when you fix a particularly gnarly bug and get to close a bunch of GitHub issues. It's not creatively satisfying, but it's rewarding all the same. And then there is all the more menial tasks like the documentation. It's not glamorous, it rarely taxes the old grey matter, but in its own way it's just as important. You could probably argue even more so and have a valid point.
I tend to think of working on Phaser as like playing an RPG. Phaser is my main character. It has attributes such as 'documentation', 'tutorials', 'community support' and 'features' that I have to keep working on improving and raising. Making sure not to leave any one of them lagging behind too far. It has a health count, which I guess is the active number of people using it. When bugs are posted it takes damage and you have to counter-act that, least it bleed out. And finally, with new versions, it levels up. It's a constant journey of self-improvement and discovery.
Who knows what will happen when we reach level 20. Maybe it'll be time to start earning epic boons :)
JSDoc to SQLite
As I mentioned at the start of this Dev Log my work on the docs this week was two-folder. Firstly, doing all the types as mentioned above. And secondly working on both my JSDoc to SQLite script and the new docs interface. JSDoc has the ability to generate a JSON dump of the documentation, rather than HTML. It's a massive unrelenting file. The Phaser JSON export is over 8MB in size.
While there are tools that exist to query JSON as if you're working with a database, mostly expanding on the JSON Pointer format, this isn't ideal because the very structure of the JSON isn't conducive to the sorts of data we wish to group together. It made far more sense, to me at least, to get the data into a database where it can be searched, queried and cross-referenced with ease. I'm a big fan of SQLite. The single file approach, combined with extensive support from both the JavaScript and PHP side (unlike, say, MySQL) makes it a perfect candidate for this. So I took the JSON to SQLite script I coded almost a year ago and finished it off. I expanded the structure of the tables, reworked the script to make it more modular and ran lots of tests. I'm pleased to say this work paid off and I've now got a work flow that allows me to quickly export the entire Phaser API documentation to JSON and then to SQLite in a matter of seconds. It also means I can ship the docs as single SQLite files along with each release.
My hope is that with the docs in this format, more tools can make use of them and grow from around them. For example, I think Phaser Editor is a good candidate for making use of this. But there's no reason why they could also form the basis of extensions for VSCode or a Sublime plugin.
The main reason of having the docs in this format is so that I can build a much better interface to them on the Phaser site. Which is what I spent the final two days of this week doing. My initial approach was to re-create the JSDoc template, just with better formatting. It was easy to get to this point thanks to a combination of using Bootstrap 4 and being able to quickly query the SQLite file and extract what is needed. I still need to spend more time refining the pages but I'm confident that I can get the docs live on the Phaser site next week. This is important for two reasons. The first, of course, is that they look and work a whole lot better than the JSDoc template. The layout is cleaner and less cluttered. You can quickly and easily navigate between sections and toggle things you don't need on and off automatically. I can also tweak the layout and keep improving it over time, without having to publish another 60MB+ to GitHub.
The second reason is actually monetary. I run small un-obtrusive ads on the Phaser site using a combination of Carbon and CodeFund. The money they generate, while not being a large amount, is still enough to cover the cost of hosting the Phaser site and also put a bit towards the general fund pot. However, up until now, the Phaser 3 docs have been hosted on GitHub pages. This is taking traffic away from the Phaser site and not counting towards ad revenue either. Both are things I need to reverse.
People find it very confusing to go to the main site and see the docs (and examples, for that matter) are all Phaser 2 based. Not even Phaser CE, but the several years old 2.6.2 release. And for anything more modern they're taken away to another site, with another style. It's just all wrong and too disparate. Moving the docs back to the site is stage 1. Stage 2 is doing the same for the examples and updating the sandbox. The final stage is a complete site redesign, for which I have lots of plans - but this is a marathon, not a sprint, so let's take it one step at a time.
Translation Help Needed
One thing I'd love to do is get the Phaser guides translated into other languages. I appreciate that translating the whole documentation is a truly massive task, so I feel that the tutorials like the Getting Started and Making your First Game are much better suited as they are shorter and more focused.
Is this something you'd be able to help with? Do you have a bit of spare time to help translate any of the tutorials to languages other than English? If so, please email me (rdavey@gmail.com).
Just let me know which language you are willing to translate from English to. Then I'll organize who does which tutorial. This is so I can make sure we don't have any overlap with more than one person doing the same tutorial in the same language.
I'm open to translations to any language, but would love if we could include Chinese or Japanese. You will be credited as the translator which can include a link to whatever site you like (your own site, a game you've released, etc). These guides have a lot of montlhy readers so it'll be brilliant to offer them in languages other than English.
What's Next
I'm going to carry on working on the docs, so I can finally get them published to the web site. Any extra time will be spent on Phaser 3.16.3. A couple of issues have arrived, including yet another with the damned fullscreen api :) which needs addressing, so I'll package up some more small fixes into another point release, hopefully towards the end of the week. As usual, check out the next dev log for further details!
Welcome to Dev Log 141. I'm going to warn you up front, this log is a bit dull :) Because I've spent the week working like mad on the documentation, which despite being vital, doesn't make for very interesting reading in a dev log.
Documentation is important for two reasons. First, and primarily, it's the thing a lot of you turn to when looking for answers. I know for a fact that a number of you go to the source code first and will manually look at a function to see what it does. But to do that, you need a good understanding of the layout of the API, and learning that takes time. So the majority of you, quite rightly, go to the API docs.
I've been working tirelessly this year on improving the Phaser 3 docs. From running the Doc Jam, to the days spent refactoring the events, to enhancing them with every release. Not a single new feature has been added to the API that isn't completely documented and it's been this way for many months now. That doesn't mean there aren't still areas that need completing - and the overall general way you browse the docs still leaves a huge amount to be desired.
The cause of this is two-fold. First, the actual docs themselves (i.e. the JSDoc blocks within the source code) need improving and expanding. There are still some areas left that don't have documentation at all, although these get less and less every month. Secondly, the HTML you browse when you visit the docs site is the direct output of a JSDoc template using the Cerulean theme. And honestly, it's far from great.
This is no fault of the template. After all, just like Phaser, it can't be everything to everyone. The overall layout, though, is just wrong. Making your own JSDoc templates is an arcane art. They're a bunch of static files that make use of a custom templating system, with HTML injected into JS injected into html. It's quite something to behold, let alone work with. The bigger issue, though, is that it takes a significant amount of time to actually generate the documentation. It will build the docs for the entire API, every single time, which generates hundreds and hundreds of files. 65MB worth each time in fact, taking several minutes. Which is less than ideal if you're making a quick tweak and want to test it to see if it looks right. Again, this isn't the fault of JSDoc. It is designed to generate static doc files from your source, that is its whole purpose. But it's not what Phaser needs.
Alert long-time readers may remember that back in Dev Log 100 I wrote about the work I was doing on the docs. At the time I was mostly talking about the app I built called Percy, which helped me generate all the docblocks the code required. However, I also alluded to the fact that I wanted to release the Phaser docs as a SQLite file and build a front-end around that. I spent a couple of days work on this at the time, but it never got any further because I was too wrapped up in actually getting Phaser itself working and the docs were written. Fast forward to this week, though, and that is where my work has been focused.
Types Ahoy
The first stage was sorting out all of the types. JSDoc offers a feature called a @typedef which allows you to create a new custom type that you then reference in the JSDocs in your code. I had been using these already, but they weren't properly namespaced, the location of the defs was inconsistent and the use of them was patchy. So the week began by addressing this.
For all areas that use them I created typedefs folders and then created a single file per type in there. For example, the Animation namespace has 7 unique typedefs at the moment, so I moved the defs out of the source code and into that folder:
Each of those JS files only contains a JSDoc typedef block. There is no code in them, so they don't add anything to the library size. Here is the typedef file for the Animation object:
As you can see it's a description of an Animation object with all of the properties listed and described. Having it like this means I can reference this type in the rest of the docs, such as in the AnimationManager.create method:
In the code above you can see that the parameter needs to be an Animation configuration object. Previously, before typedefs, lots of methods this like would have just said 'object' as the type. With typedefs you now know the exact shape of the object it expects. This should prove especially useful for TypeScript users too.
Of course, creating all of these types was a lot of work. And it was that mind-numbing labor-intensive type of work, too. Absolutely zero fun, zero creativity, no challenge. Just hour after hour, day after day, of cutting and pasting. The things I do for you guys :) Ultimately, it was worth it. It instantly made the JSDocs better. Even using the current lacklustre template it improves it hugely, because types are organized into a single place per namespace, and you can click them when viewing a method that uses one as a parameter and get taken right to the definition for it.
So, time well spent, despite the monotony of it. I find this to be a recurring trend when it comes to Phaser, or I guess any large scale open source project. There's the fun creative side of it, like adding cool new features such as improving something in the renderer. There's the challenging 'triumphant' side of it when you fix a particularly gnarly bug and get to close a bunch of GitHub issues. It's not creatively satisfying, but it's rewarding all the same. And then there is all the more menial tasks like the documentation. It's not glamorous, it rarely taxes the old grey matter, but in its own way it's just as important. You could probably argue even more so and have a valid point.
I tend to think of working on Phaser as like playing an RPG. Phaser is my main character. It has attributes such as 'documentation', 'tutorials', 'community support' and 'features' that I have to keep working on improving and raising. Making sure not to leave any one of them lagging behind too far. It has a health count, which I guess is the active number of people using it. When bugs are posted it takes damage and you have to counter-act that, least it bleed out. And finally, with new versions, it levels up. It's a constant journey of self-improvement and discovery.
Who knows what will happen when we reach level 20. Maybe it'll be time to start earning epic boons :)
JSDoc to SQLite
As I mentioned at the start of this Dev Log my work on the docs this week was two-folder. Firstly, doing all the types as mentioned above. And secondly working on both my JSDoc to SQLite script and the new docs interface. JSDoc has the ability to generate a JSON dump of the documentation, rather than HTML. It's a massive unrelenting file. The Phaser JSON export is over 8MB in size.
While there are tools that exist to query JSON as if you're working with a database, mostly expanding on the JSON Pointer format, this isn't ideal because the very structure of the JSON isn't conducive to the sorts of data we wish to group together. It made far more sense, to me at least, to get the data into a database where it can be searched, queried and cross-referenced with ease. I'm a big fan of SQLite. The single file approach, combined with extensive support from both the JavaScript and PHP side (unlike, say, MySQL) makes it a perfect candidate for this. So I took the JSON to SQLite script I coded almost a year ago and finished it off. I expanded the structure of the tables, reworked the script to make it more modular and ran lots of tests. I'm pleased to say this work paid off and I've now got a work flow that allows me to quickly export the entire Phaser API documentation to JSON and then to SQLite in a matter of seconds. It also means I can ship the docs as single SQLite files along with each release.
My hope is that with the docs in this format, more tools can make use of them and grow from around them. For example, I think Phaser Editor is a good candidate for making use of this. But there's no reason why they could also form the basis of extensions for VSCode or a Sublime plugin.
The main reason of having the docs in this format is so that I can build a much better interface to them on the Phaser site. Which is what I spent the final two days of this week doing. My initial approach was to re-create the JSDoc template, just with better formatting. It was easy to get to this point thanks to a combination of using Bootstrap 4 and being able to quickly query the SQLite file and extract what is needed. I still need to spend more time refining the pages but I'm confident that I can get the docs live on the Phaser site next week. This is important for two reasons. The first, of course, is that they look and work a whole lot better than the JSDoc template. The layout is cleaner and less cluttered. You can quickly and easily navigate between sections and toggle things you don't need on and off automatically. I can also tweak the layout and keep improving it over time, without having to publish another 60MB+ to GitHub.
The second reason is actually monetary. I run small un-obtrusive ads on the Phaser site using a combination of Carbon and CodeFund. The money they generate, while not being a large amount, is still enough to cover the cost of hosting the Phaser site and also put a bit towards the general fund pot. However, up until now, the Phaser 3 docs have been hosted on GitHub pages. This is taking traffic away from the Phaser site and not counting towards ad revenue either. Both are things I need to reverse.
People find it very confusing to go to the main site and see the docs (and examples, for that matter) are all Phaser 2 based. Not even Phaser CE, but the several years old 2.6.2 release. And for anything more modern they're taken away to another site, with another style. It's just all wrong and too disparate. Moving the docs back to the site is stage 1. Stage 2 is doing the same for the examples and updating the sandbox. The final stage is a complete site redesign, for which I have lots of plans - but this is a marathon, not a sprint, so let's take it one step at a time.
Translation Help Needed
One thing I'd love to do is get the Phaser guides translated into other languages. I appreciate that translating the whole documentation is a truly massive task, so I feel that the tutorials like the Getting Started and Making your First Game are much better suited as they are shorter and more focused.
Is this something you'd be able to help with? Do you have a bit of spare time to help translate any of the tutorials to languages other than English? If so, please email me (rdavey@gmail.com).
Just let me know which language you are willing to translate from English to. Then I'll organize who does which tutorial. This is so I can make sure we don't have any overlap with more than one person doing the same tutorial in the same language.
I'm open to translations to any language, but would love if we could include Chinese or Japanese. You will be credited as the translator which can include a link to whatever site you like (your own site, a game you've released, etc). These guides have a lot of montlhy readers so it'll be brilliant to offer them in languages other than English.
What's Next
I'm going to carry on working on the docs, so I can finally get them published to the web site. Any extra time will be spent on Phaser 3.16.3. A couple of issues have arrived, including yet another with the damned fullscreen api :) which needs addressing, so I'll package up some more small fixes into another point release, hopefully towards the end of the week. As usual, check out the next dev log for further details!