Rearrange structure of the media repository
Remove the separation between art and final assets, so that it becomes both clearer how the assets fits together, as well as easier for content creators to create new content. Instead of the current separation between source assets in "art" and final assets in both "3d_objects" and "3d_skeletons" we should keep all of the assets closer to each other.
This is done by combining them into the "assets" directory.
Some rules:
Directories always use lower case characters, never upper case. This is to prevent issues when dealing with systems which don't differentiate between upper and lower case (i.e. Windows and OSX in some settings).
Final textures however always use upper case names, i.e. "D.png", "S.png" etc. This is mainly for legacy reasons. This also applies to the AUTHORS and README files, or any other license files.
The general directory layout is thus:
assets/
assets/
assets/
assets/
The final directories are defined thus:
<asset name>
This should be named as close to the actual asset as possible. In pretty much all cases this is no issue since this will come naturally. An axe will be named "axe", a wolf will be named "wolf" etc.
This directory should also contain a copyright attribution file, named AUTHORS.
Optionally it could also contain a README file with more information about the asset.
We are also considering putting the Ember ".modeldef" files here, instead of having them in the client source.
<asset name>/source
All sources used for creating the asset are put here. This would be Blender files, PSD files etc.
<asset name>/model
The client ready models would be put here. This would be Ogre mesh files.
<asset name>/texture
Client ready textures should be put here. These follow this strict naming scheme. Note that we here describe the PNG versions. We want however to start using DDS textures, so we might revise this in the future to instead use ".dds" extensions instead of ".png":
D.png : A diffuse texture, either with rgb or rgba.
S.png : A specular texture. Always grayscale, no alpha.
N.png : A normal map, with heights. The normals are put in rgb, and the height in the alpha channel.
I.png : An icon, rgba, always square and with size fitting power of two (128, 256, 512 etc.)
AO.png : (Not currently implemented in client) Ambient occlusion map. Grayscale, no alpha.
<asset name>/sound
Client ready sound files. In OGG format.
The top level directories are defined thus:
assets
Top directory for all game assets, i.e. stuff that should be shown in a client as part of the world. The child directories contained here denote the different types of settings for the assets, or "worlds". We currently only provide the medieval "Dural" setting, but expect there to be other settings going forward (perhaps a modern setting, or a space setting).
assets/dural
All of our current assets are suited for the medieval "Dural" setting, and should therefore be put under this directory.
assets/
The top level category should be abstract enough to encompass a lot of different types of assets, but at the same time clear enough that it's intuive where a certain asset should belong. Under each category there can then be multiple levels of subcategories, depending on the need.
assets/dural/items
Items of all types, i.e. things that can be used by the characters in the world. Or saw, or an axe for example.
assets/
Creatures of all types.
assets/dural/plants
Plants of all types.
assets/
Things that belong to the environment, like boulders, skies, various types of ground.
assets/
Things that are built, like houses, walls etc.
assets/
Constructs that doesn't really fit in the world, but are used by the client, perhaps as placeholders or when authoring the world.
Below a category or subcategory we then place the concrete directory for the asset. This is where the directories "sources", "models", "textures" and "sounds" are placed. In many cases these directory are adjecent, but they mustn't always be, as we'll show with an example.
To start with a simple example, consider the "hatchet" asset. This is a fairly simple mesh of a hatchet with textures to go along.
All assets would thus be put in
assets/
The complete listing would look like this:
assets/
assets/
assets/
assets/
assets/
assets/
assets/
assets/
assets/
assets/
assets/
assets/
If there are sounds to go with the hatchet, the client ready versions would be put in assets/
Sometimes however there are multiple meshes or textures that go along with one asset. As long as the asset logically can be considered as one unit, these should then be placed slightly differently depending on whether they are meshes or textures.
So, consider that we should add a version of the hatchet which is broken. This would require a new mesh, which would then go into:
assets/
Consider then that we want to add a new texture of a rusty hatchet. Since the textures use the generic names (D.png etc.) we must put this in a separate directory. It would thus be:
assets/
We allow flexibility here, since the most optional structure would differ for each asset. We could very well also consider the "whole" and "rusty" textures from the beginning, and then instead have created
assets/
assets/
from the start. Either approach is just as valid, as long as the general structure is applied.
This also applies to template textures. While we don't envision too many new assets using templates being created (since it's not always a good fit for all meshes, and the overhead of handling the template makes the authoring process cumbersome) we do allow for them to fit within the new structure.
Consider the tools then, and the "tool handle" template texture. It would thus be put at
assets/
The source for the "tool handle" template would thus be put in assets/
A note on copyright attribution and licensing.
Whenever any asset is licensed using a license differently from the default ones (which are specified at the root of the repo) the licensing files that specifies the license must be included. They can be named a little differently depending on the license. Like COPYING, or License.txt etc.. These should be put where they are most applicable. The asset item directory is often most suitable, but this might not always be the case.
Likewise with AUTHORS. While the asset item directory often is most suitable, this is not strict. If there are additional authors for a specific file or directory additional AUTHORS files can be freely placed anywhere. Don't overdo this though, since updating AUTHORS files adds to the overhead. The Subversion history should also contain all information regarding authors that's needed.
Legacy sources
There will probably be some sources that won't fit into the "assets" structure. I can't say what these would be currently, but we're bound to find some variants. The best way to handle these currently would be to just put them in the top level "sources" directory (the one currently named "art"). Depending on how many they are we then need to process them separately; either moving to the junkyard or otherwise better incorporate them.
Blueprint information
- Status:
- Complete
- Approver:
- Erik Ogenvik
- Priority:
- Medium
- Drafter:
- Erik Ogenvik
- Direction:
- Needs approval
- Assignee:
- Dean Bouvier
- Definition:
- Approved
- Series goal:
- None
- Implementation:
-
Implemented
- Milestone target:
- None
- Started by
- Dean Bouvier
- Completed by
- Erik Ogenvik
Related branches
Related bugs
Sprints
Whiteboard
In regards to #1 - Do you think it makes sense to separate things at the root of the wf_trunk? So perhaps a wf_trunk/
Regarding #3 - Naming and directory structure needs to be made universal. 3d_objects is well ordered and follows a set pattern which the client utilizes. In art there is some attempt to follow the same pattern in a few places, but at least 2 other methods are used as well. For art I am not sure that having the models and textures separate later in the structure is advantageous. If all models used unique textures it would make sense to nest them inside a common directory with the mesh, but because we want to promote texture templates it doesn't make sense to do this. Perhaps breaking the art into models and textures at the very start would work best.
Yeah, it makes sense to put the game type first. I.e. "mason/3d_objects" and "mason/art".
/erik
I've visualized the restructuring using the confluence site I've been working with and testing out.
I've also added some suggested additional plans. Please take a look at it and give me your thoughts.
http://
/Dean
Sorry I didn't respond earlier. My first issue is with introducing capital letters. Wouldn't it just be better to follow the existing naming structure and use lower case letter throughout?
I'm not too fond of the name "games" either. I as I've thought about it I don't think we should use the name "mason" neither, as that's just the ruleset. There are many rulesets in the same world ("acorn" and "mason" being the current ones). In addition, I'll pretty soon push for a move from "mason" into a new ruleset name, as the direction of the project currently differs very much from what's envisioned in "mason".
Instead I would like to use the name of the setting, or the world, which is "dural". See http://
I would therefore suggest a replacement of "Games/Mason" with "worlds/dural".
Another issue which I see now that I've forgot to add to the requirements is that I want to make sure that the structure allows for an automatic content pipeline. Basically that artists should be able to create stuff in the "art" directory and have some kind of process automatically generate the client assets (under 3d_objects). This structure would allow for this, but it also hinges on making sure that the underlying "art" and "3d_objects" structures are synchronized.
/Erik
Yep I totally agree, that was more a matter of carelessness then design, definitely makes sense to keep it all lower case. Good point though.
I also think your right about the usage of worlds/dural, it makes a lot more sense, also it won't need to change when next stage is entered, and provides a more clear example for how other worlds would fit in alongside other world assets.
Sean and I have been discussing this task recently and we both feel that its important that the new structure lend itself to future dynamic media plans, and I think your plan of having an automatic content pipeline is very good as well. I have long felt that the art should mirror the game ready layout as much as possible, it avoids confusion for artists in navigating or adding new content.
Currently I have "art" directory inside the rulset, and it could be considered the mirror root of 3d_objects as it currently does more or less. Another option is to have art be the mirror root of "worlds" so it would have "/art/dural/
Another approach could be to create a "common" section which would have things like resources, themes, and an art directory that mirrored those but contained original formats, That could be used for non 3d_objects and sounds which are not really world assets, but rather global game client assets.
/Dean
Yeah, it sounds more sensible to put the "art" directory one level up, as a sibling to "worlds". That way it would also be easier to implement a "common" subdirectory, if it turns out that's needed.
/Erik
Ok I made the changes we've discussed on the page.
One thought I just had was perhaps it would make sense to rename "art" to "originals" to more appropriately reflect its use.
Also do you agree that the junkyard could be cleaned up and archived, or should we just leave it alone? There are some duplicates and game used assets that will need to be addressed though I believe.
/Dean
I don't know the idiomatic nomenclature for designers, but I agree that "originals" (or "source") is a more proper name than "art", given its purpose.
Just keep junkyard; it's always good to have as a reference.
/Erik
Indeed, I think "source" is a very clear indicator of its use. Having a clear separation between these but mirroring the structure should help a great deal. I'll update the proposal to reflect these things.
/Dean
I think it would be good if the structure helped facilitate or reflect texture template use. For instance inside creatures I think it makes sense to further structure them by the template for instance /creatures/
Also does it make sense to have a "characters" section? These are really just creatures who in the ruleset have the playable variable set to 1. Most of the current character types would fit nicely into /creatures/
Not all meshes will be suitable for templates and some like structures will likely end up using multiple template types for different parts. I'd like to recommend for any like those there be a templates.txt file added to the source location which defines it as being unique or lists all the various templates it makes use of. This could provide a much faster method of discovery for artists or world builders who want to try variations but are unsure of what to search for.
/Dean
I agree that there's no reason to keep the "characters" section.
On the template issue, while I agree that there should be some way to see what template to use, I'm not so sure if it's a good idea to hard code that into the directory hierarchy. Putting it into the path promotes it to a significant place, and there's a risk that it will be too restrictive. I can think of cases where you would want to use a specific template in multiple places, for meshes which don't really have anything else in common.
/Erik
Yes, I've used some odd template textures myself for some of my models that would not be obvious to others without open the source and looking. I am thinking for builders it would be nice to have some form of tagging or filtering possible. but you're right perhaps group in the directory structure is not the best way to achieve this.
It would make sense to organize some areas like "creatures" and "weapons" down one more level I think, they have possibility to become huge sprawling directories over time and the only way to make searching for a common type like finding all the "fish" or "bow" would be to prefix them with that.
Having all sections be divided by "models" and "textures" should be a consistent structure and make automated game assets much cleaner. Currently only some "source" sections do this. I think doing this along with grouping things together and maybe just adding a "templates.txt" file to the area that lists commonly used templates might really help in promoting template usage.
In some cases source sections have "readme.txt" files that explain the purpose of that area, on thinking over this, it might be best to just add templates used to this file and just add more as needed.
I've made some changes, to hopefully clarify what I'm discussing here about the types and categorizing. I think we are pretty close on having the structure ironed out I think. If you agree I would like to begin working on things.
/Dean
The current proposal looks very good. I'd say we should go for it now. Are there any areas where you still think we should think things through some more?
/Erik
I think it's a good plan and covers things well. I'll be begin laying out the various work items for the task.
/Dean
I've thought about this a little more and have come to the conclusion that we probably instead want to remove the "art" directory and move all of the source material into the "worlds" directory, put in directories named "source" right alongside the "models" and "textures" directories. I also want to try to categorize all assets by their nature, rather by their technical type. Therefore I also would want to move all sounds to be alongside "models" and "textures".
So, if we consider the pros and cons of having the "art" directory separate as it is currently, the only upside as I see it is that it's possible to check out the "3d_objects" or "3d_skeletons" directories independently without having to check out "art". However, I can't see any real use cases for this.
Any asset creator would without doubt want to check out the complete repository. In addition, while it some years ago could be considered "large", that can't be said today with broadband and 3TB disks readily available.
Any regular user who only wants to get the final asset wouldn't use the repository anyway; he or she would use the pre-prepared assets as provided on amber or through our media tar files.
The downsides are many however. Most obvious is that it makes everything more confusing and harder to overlook. If you're working in one directory in "art" it's not always easy to find where the corresponding generated assets ends up.
However, one issue that's probably more serious is that it makes assets creating much harder, especially any process where we both want server side automation and allowing for anyone to create 3d assets independent of the repository (I'll get back to what I mean by that).
To exemplify I'll walk through the kind of workflow I envision.
But first I'll just enumerate the changes I propose:
* Instead of the top level directory of "worlds" I want a top level directory "assets". This will contain all models, textures, sounds and source materials.
* Just like the current proposal I want the second level to denote the world it belongs to, currently only "dural".
* The third level should then be "creatures", "environment" etc. (i.e. what's currently on the fourth level in the proposal)
* Alongside "models" and "textures" there should now be "source" directories which contain the source files used for creating whatever is in the aforementioned "models" and "textures" directories. Basically we'll move what's currently under "art" to where it should be under "assets".
* When we generate the media tarball it's then easy just to ignore whatever is under the "source" directories. Same with Ember; it's easy to just add some logic to the asset loading code to ignore "source" directories (i.e. so you can point Ember to the raw repo)
Ok, so now our example.
Say we want to create a new asset, a vase. We do this by first creating a new directory "assets/
Once we feel we're done with applying the texture we then need to export the mesh. This can be done in various ways. We can either do it manually. The mesh would then be placed in "assets/
However, we can also automate all of this, much thanks to the source being close to the final exported assets.
I we craft an exporter (probably in Python) it can easily take the "assets/
As I mentioned earlier this also makes it easier to allow for more detached assets creation. All of the various conventions of where to place assets might be a little overwhelming for new assets creators who only want to add some simple assets without even having to check out the whole repository first. For them I'm envisioning this workflow:
Say someone new to the project wants to add a vase, just like in the previous example. However, they don't have access to the repository. Instead they just follow a few simple guidelines and then use a web tool we provide. The guidelines are to use the two directories: "source" and "textures".
In an arbitrary place on their local machine they create a "vase" directory. Within this they create "source" and "textures" directories. Under "source" they place their "vase.blend" and their "vase.psd|xcf" file. Under "textures" they place the "D.png" file which is generated from their "vase.psd|xcf" file. Once everything is in order they create a zip file of everything, "vase.zip". They then go to a simple web tool we provide (WOMBAT perhaps?), they go to the most suitable place in the hierarchy for their asset "dural/
With this setup I'm thinking that the process should be simpler both for existing assets creators as well as any new ones.
/Erik
Work Items
Work items:
Move "3d_objects" and "3d_skeletons" into the common "assets/dural" directory: INPROGRESS
Organize all game assets by models and textures structure: INPROGRESS
Make sure that all assets follow the correct naming scheme: INPROGRESS
Organize models and textures by common type where applicable: INPROGRESS
Move "art" assets into the "assets/dural" directory, at proper locations: TODO
Update all Ogre materials with proper names after moving them: TODO
Update all Ogre meshes with new material names: TODO
If any "art" assets remain that couldn't fit into the "assets/dural" directory, rename "art" to "sources": TODO
Make sure all assets are correctly attributed with "AUTHORS" files: TODO
Dependency tree
* Blueprints in grey have been implemented.
