2023 Archives Progress

November 1

As mentioned in the Armory Progress thread, the Armory has mostly been in limbo since October 22nd. I have not wanted to change much on it during the transition to going online, to keep the amount of variables down. In that time I have been working on overhauling the server-side code of the Archives to try to make it functional again, and have had spectacular results. Before that though, a recap is in order since this project is OLD.

The Armory was last online in 2016, and for a large portion of its existence, on the nav bar were links to the Forums and to the Archives. These forums came to exist toward the end of the Armory's time online, and the Archives never manifested at all due to the scale of the challenges involved in making it possible.

The Archives were meant to serve the general purpose of a wiki for Vindictus, but with a single profound difference: Instead of being user edited, I wanted to auto-generate all of the pages based on data in my database. Naturally 100% auto-generation is simply impossible, I would have to use something like ChatGPT to produce descriptions and explanations for things like the Reforging system (ChatGPT didn't exist back then) and the results would be...well you've seen examples of ChatGPT stuff likely. The compromise I reached was that I would auto-generate all associations between items and other things, and manually write the descriptive part of each page. Only a minority of pages even need text actually, just informational pages, so I could still auto-generate and auto-update the vast majority of the project.

I started work on actually making a functional piece of the Archives in 2021, code to generate pages for items. I made reasonable progress at the time. I had a working general layout for item pages, I could search for items within the Archives itself, and I could link them in human legible links. Items could show their stats, requirements, and properties such as if they were tradable or would bind when traded. Items could also show the recipe involved in crafting them, their drop locations, and what they could dismantle into.

The problem is that this functionality came from a structural setup for the Armory that I wasn't 100% confident in the ToS-friendliness of. Only in 2022 I think it was did I come upon a way to make the Armory feasible that was 100% certain to be acceptable within Nexon's ToS. I took the opportunity with this forced restructuring to completely rebuild my databases into a more optimal format, which did pay massive dividends when rebuilding the Armory to use them. Unfortunately, this meant literally everything the Archives looked to access no longer existed. In a lot of cases, there straight up wasn't even something similar to point it to. I had believed it to be a huge undertaking to restore the Archives to use my new setup.

This last week has proven me wrong. All of the previous functionality is restored, plus one new functionality I had also sorely wanted for the Archives. One thing I really appreciate about RuneScape's wiki, aside from Jagex's support and contributions to it, is that at the bottom of every page it has links to relevant related pages. An article about Rune Longswords would give you links to other Longswords, etc. I really wanted something like this on the Archives, since it saves a lot of navigation when looking for similar things. I now have a working generator for these, though the code involved is absolutely hideous looking for now. It will be polished once I have ironed out all of the kinks with different types of items.

There are still significant hurdles I need to deal with, even ignoring that I have a huge amount of data entry ahead of me on this thing. First, the search results I get are in an almost useless order, and I will have to do quite a bit of studying to learn how to make a proper search. That's even before factoring in that ontop of Item pages, there will also be pages for Battles, Enemies, Information, and probably other types yet undrafted.

I also need to figure out how to manage/update these once the Archives are online. It is implausible to generate them on the fly every time someone requests them, the amount of work to generate the page is rather sizable. They need to be pre-generated, but I also have to be able to dynamically update all relevant pages when something is added or changed. For example, due to the related links at the bottom, adding a new Orna weapon for a new character requires updating every Succession Ore page, the Orna's Essence: Weapon page, every Orna weapon's page, each Orna armor piece's page, every raid battle's page, every raid boss's page, (maybe) Dianann's page, and (maybe) the page for the expertise of whichever one makes the weapon.

Problems aside, here are some images of the current build and let's look at what we have so far:



This is what weapon/armor pages will look like, for the most part, and these are in the most complete state so far. On the left you have the links to various sections of the Archives. At the top of the right-side you have the search bar, to navigate the Archives more deliberately. On the right below the search bar, you have an overview of the item's properties. On the left in the main section, you have the title, followed by any text description and details about the item. Item Details shows the stats, requirements, and other important details about the item. Then you have the dynamic sections, such as crafting materials, what it can be used to craft, what it dismantles into, where it drops, etc.

Peeking out at the bottom of this image (cropped due to running out of page height, you have the related links for this item. The sections are default collapsed, but can be expanded by clicking. As this is an Orna Hasta, it is showing links to Orna Equipment (the armor pieces, the offhands, and all weapons), and all other Hasta. It does not show related links to the Orna materials, since they are already there in the crafting data. I am making an effort to try to avoid redundant links in this section in order to keep the size down and keep usability high.



Here we have the search bar. For items, it will show you the name and icon of each result. The search bar currently only shows a max of 15 results. I haven't decided on a display format for any page type except items yet.



In the final image of this log entry, we have the page for the Lament Enchant Scroll. Here you can see that the details shown are different for enchant scrolls, to be contextually relevant to the item. You can also see the current draft of the layout for drop tables, mostly the same as it was in 2021. The only difference is that I switched out the small circles for shapes closer to matching the Armory UI. The blocks are lit when it drops in the relevant mode. Blue is Solo, Yellow is just a regular run of the battle, and Red is Hero mode.

Additionally, you can see in this pic that the Archives has seen that this item drops in at least one location, and so is showing the detailed information for each drop location. This will only be the case for items that drop in 6 or fewer locations; I haven't decided what form to abbreviate the data in for items that drop in more locations, such as pigments.

This post is mostly just to give a refresher on the Archives, the next post will actually be me going into a decent bit of detail on how the Archives functions. This next post will likely not be a development update, but more of a part 2 to this post. It will probably be very soon, but no promises.

November 1 (Continued)

As promised in the Dev Log earlier today, this one will be on the more technical side of how the Archives currently works. Likely in far more detail than I ever explained the workings of the Armory.

Navigation

I tend to design things on a whim based on what I feel most motivated to work on, so the starting point of the entire project, for me, was figuring out how to link the pages to others. For the Archives to be truly useful, this was a must. The problem was, I had no experience with this kind of thing. The Armory stored data in its shareable set links, but it was not human legible in the slightest. I wanted the link to the page on the Archives to capable of describing what the page was, with no uncertainty.

It was at this point I had a resolution that I wanted the links to contain the name of the item the page was for. There was no way to be more clear than literally having the name right there. The problem comes in the fact that names are not uniquely identifying. At the time I was pushing this, for example, there were around six Regina's Feather: Weapon, each with very different properties. One was Fine rank with low stats and caps, four or five of them were Superior rarity with different caps on enhancement/quality, and one was the Rare rarity material we still have to this day. If you linked to /Item/Regina's_Feather%3A_Weapon, how would the Archives know which to load?

I actually still haven't fully resolved this issue to my satisfaction even now. For composite materials it is of course no longer a concern. However, if you were to go to /Item/Special_Raider's_Pants, there are two items named that. One for males, and one for females. On that page you get a drop down list to select variants, and it defaults to the male version (just since it came first by numeric ID in the database). You can use the drop down to switch to the female version, or you can link to /Item/Special_Raider's_Pants/1, but this is obviously not as clear as the rest of the link. I suspect this problem will haunt me for most of development.

Page Content

This is the bulk of the auto-generated content based on the data in the Armory's database. The Armory recognizes if items are composite or not, so the Archives can too, and so on. What this does not get me is the extra information, such as any drop locations, if it can be crafted, or if it dismantles. For these, simply enough, I just made additional tables for that data; though unfortunate this also means I have to fill them in.

The particular one I feel clever about here is the table for crafting. Since Expertise no longer saves materials vs NPC crafting, I can represent NPC Crafting and Expertise Crafting with a single table. I just need a pair of flags to indicate if each one is valid for that item, as part of each item's record in the table.

Related Links

This one was hard to conceptually figure out, but actually fairly easy in implementation. I can infer some connections off the Armory data, such as items that are part of the same set bonus being related. However that breaks down entirely for outfits, and doesn't properly cover composite items. "Regina" through that would match the five armor parts, and all of the weapons, but none of the offhands since they are not part of the set bonus. "Peony Fox Outfit" through that would match every single outfit piece in the entire game since they were given the same set data for simplicity in the Armory.

Ultimately, after iterating a few trials, I came upon two differing approaches for this section, each covering different types of relational links. First, at the top of the section, are auto-generated relations, such as what you see for the Lament Enchant Scroll in the previous entry. A flag is set that it is droppable, so it is pulling and listing the drops of every battle it can drop in (if below 7 matches). Something like this will also happen for Reforging and Enhancement, when the Archives can process those.

The second approach is a far more manual undertaking. I created a new table where I could apply a list of tags to each item. These tags have their own hierarchy, where one tag "contains" several lower tags, and so on down the stack. Each tag has a defined name, a flag for if it should appear in Related Links, a flag for if all sibling tags should be listed in Related Links (such as all Weapons being limited to ONLY the relevant weapon for the page), and a value for how to sort the contents of that tag's section.

For example, the Keen Succession Ore is tagged as an Orna weapon material, and as an invisible Orna equipment. The Orna weapon material tag paths up one step to "Orna Materials", it is set to fully show, so it lists all materials of all type, separated by their individual tags. The invisible Orna equipment tag paths up one step to "Orna Equipment", but is set to hide itself; so for this one, all Orna equipment items are shown, separated by individual tag, but the material's invisible tag does not show. It solely exists to indicate the relation.

To prevent recursion, I have limited traversal on the hierarchy to a single step upward or downward. This should also promote more robust relations between items through better tagging. I wouldn't call this piece of the Archives 100% yet, but I am liking how it is turning out so far.

Page Types

Naturally, not everything in the game is an item. These different things need different page formats, so they are built with entirely separate code and HTML templates. This one is last in this post because I have not even conceptually drafted any other page types, except to acknowledge they will exist. A link to an Orna Hasta would look like [site name]/Item/Orna_Hasta, and a link to Milo's raid would look like [site name]/Battle/Dialectic_of_Death.

This one is sure to cause me headaches in code structure and search results in future, but it is the easiest route to make things work overall.