PAK (Metroid Prime) - Revision history

>Aruki: /* Tools */

2019-04-13T01:31:46Z

Tools

← Older revision		Revision as of 01:31, 13 April 2019
Line 161:		Line 161:
	== Tools ==		== Tools ==

	* [https://~~drive~~.~~google~~.com/~~file~~/d/~~0B9MLV21H7SDvcGc1QjRfSTFJNk0/view~~?~~usp~~=~~sharing~~ PakTool] by Aruki		* [https://www.dropbox.com/s/bba2n9jzj719by4/PakTool.rar?dl=0 PakTool] by Aruki

	[[Category:File Formats]]		[[Category:File Formats]]
	[[Category:Metroid Prime]]		[[Category:Metroid Prime]]
	[[Category:Metroid Prime 2: Echoes]]		[[Category:Metroid Prime 2: Echoes]]

>Aruki: /* Named Resources */

2018-06-14T19:54:42Z

Named Resources

← Older revision		Revision as of 19:54, 14 June 2018
Line 73:		Line 73:
	\| ''Name Length''		\| ''Name Length''
	\| '''Name String'''		\| '''Name String'''
	\| Name of the asset. This name usually corresponds to a hardcoded string that the game uses to look up the asset, so this generally can't be changed~~. Not zero-terminated~~.		\| Name of the asset. Not zero-terminated. This name usually corresponds to a hardcoded string that the game uses to look up the asset, so this generally can't be changed.
	\|-		\|-
	\| colspan=5 {{unknown\|End of entry}}		\| colspan=5 {{unknown\|End of entry}}

>Aruki: /* Named Resources */

2018-06-14T19:54:29Z

Named Resources

← Older revision		Revision as of 19:54, 14 June 2018
Line 73:		Line 73:
	\| ''Name Length''		\| ''Name Length''
	\| '''Name String'''		\| '''Name String'''
	\| Name of the asset. This name usually corresponds to a hardcoded string that the game uses to look up the asset, so this generally can't be changed.		\| Name of the asset. This name usually corresponds to a hardcoded string that the game uses to look up the asset, so this generally can't be changed. Not zero-terminated.
	\|-		\|-
	\| colspan=5 {{unknown\|End of entry}}		\| colspan=5 {{unknown\|End of entry}}

>Aruki: /* Format */

2018-06-14T19:53:23Z

Format

← Older revision		Revision as of 19:53, 14 June 2018
Line 15:		Line 15:
	{\| class="wikitable"		{\| class="wikitable"
	! Offset		! Offset
	! ~~Size~~		! Type
	! ~~Description~~		! Name
			! Notes
	\|-		\|-
	\| 0x0		\| 0x0
	\| 4		\| int16
	\| '''Version ~~number~~'''. Always ~~0x00030005.~~		\| '''Version Number Major'''
			\| Always 3
			\|-
			\| 0x2
			\| int16
			\| '''Version Number Minor'''
			\| Always 5
	\|-		\|-
	\| 0x4		\| 0x4
	\| 4		\| int32
	\| '''Unused'''~~; always~~ 0		\| '''Unused'''
			\| Always 0
	\|-		\|-
	\| 0x8		\| 0x8
	\| colspan=2 {{unknown\|End of header}}		\| colspan=3 {{unknown\|End of header}}
	\|}		\|}

Line 38:		Line 46:
	{\| class="wikitable"		{\| class="wikitable"
	! Offset		! Offset
	! ~~Size~~		! Type
	! ~~Description~~		! Count
			! Name
			! Notes
	\|-		\|-
	\| 0x0		\| 0x0
	\| 4		\| {{FourCC}}
	\| '''~~File type~~''' ~~fourCC~~.		\| 1
			\| '''Asset Type'''
			\| Indicates the type of asset (texture, model, area, etc) and usually doubles as the asset's cooked file extension.
	\|-		\|-
	\| 0x4		\| 0x4
	\| 4		\| int32
	\| '''~~File~~ ID'''.		\| 1
			\| '''Asset ID'''
			\| Unique identifier for this asset.
	\|-		\|-
	\| 0x8		\| 0x8
	\| 4		\| int32
	\| '''Name ~~length~~''' ~~(NL)~~		\| 1
			\| '''Name Length'''
			\| Length of the name string
	\|-		\|-
	\| 0xC		\| 0xC
	\| NL		\| char
	\| '''Name'''~~; not zero-terminated~~		\| ''Name Length''
			\| '''Name String'''
			\| Name of the asset. This name usually corresponds to a hardcoded string that the game uses to look up the asset, so this generally can't be changed.
	\|-		\|-
	~~\| 0xC + NL~~		\| colspan=5 {{unknown\|End of entry}}
	\| colspan=2 {{unknown\|End of entry}}
	\|}		\|}

Line 67:		Line 84:
	{\| class="wikitable"		{\| class="wikitable"
	! Offset		! Offset
	! ~~Size~~		! Type
	! ~~Description~~		! Name
			! Notes
	\|-		\|-
	\| 0x0		\| 0x0
	\| 4		\| int32
	\| '''Compression ~~flag~~'''~~; this~~ will either be 0 or 1, with 1 denoting a compressed ~~file~~.		\| '''Compression Flag'''
			\| Value will be either 0 or 1, with 1 denoting that the asset is compressed.
	\|-		\|-
	\| 0x4		\| 0x4
	\| 4		\| {{FourCC}}
	\| '''~~File type~~''' ~~fourCC~~.		\| '''Asset Type'''
			\| Indicates the type of asset (texture, model, area, etc) and usually doubles as the asset's cooked file extension.
	\|-		\|-
	\| 0x8		\| 0x8
	\| 4		\| int32
	\| '''~~File~~ ID'''		\| '''Asset ID'''
			\| Unique identifier for this asset. This ID is used by other assets to reference this one.
	\|-		\|-
	\| 0xC		\| 0xC
	\| 4		\| int32
	\| '''Size''' ~~(note:~~ always ~~a multiple of~~ 32. The end of the ~~file~~ is padded with 0xFF ~~in order to be 32-byte aligned~~.)		\| '''Size'''
			\| Size of the asset data in the pak. This is always 32-byte-aligned. The end of the asset is padded with 0xFF.
	\|-		\|-
	\| 0x10		\| 0x10
	\| 4		\| int32
	\| '''Offset'''		\| '''Offset'''
			\| Offset of the asset data within the pak. This is an absolute offset.
	\|-		\|-
	\| 0x14		\| 0x14
	\| colspan=2 {{unknown\|End of entry}}		\| colspan=4 {{unknown\|End of entry}}
	\|}		\|}

>Aruki: /* Compression */

2018-06-14T19:46:34Z

Compression

← Older revision		Revision as of 19:46, 14 June 2018
Line 118:		Line 118:
	* [[CRSC (File Format)\|CRSC]]		* [[CRSC (File Format)\|CRSC]]

	Additionally, in Prime 2 any files can be left uncompressed if the compressed file is larger than the uncompressed one. This is not the case in Prime 1 (although custom repacking implementations should probably do this regardless).		Additionally, in Prime 2 any files can be left uncompressed if the compressed file is larger than the uncompressed one. This is not the case in Prime 1 (although custom repacking implementations should probably do this regardless, as there's no benefit to the compression in this scenario).

	== Optimizing Load Times ==		== Optimizing Load Times ==

>Aruki: /* Optimizing Load Times */

2018-06-14T19:44:23Z

Optimizing Load Times

← Older revision		Revision as of 19:44, 14 June 2018
Line 131:		Line 131:
	* First the game goes down the area dependency list in the [[MLVL (File Format)\|MLVL]] file and loads every file in the list for every active layer. The game follows the exact order specified by the list.		* First the game goes down the area dependency list in the [[MLVL (File Format)\|MLVL]] file and loads every file in the list for every active layer. The game follows the exact order specified by the list.
	* After loading all area dependencies, the area itself is loaded.		* After loading all area dependencies, the area itself is loaded.
	* Finally the game loads assets being used by the new area that are missing from the list. This primarily means [[SCAN (File Format)\|SCAN]] files (in MP1), the skybox model, and assets being used by PlayerActor animsets (this avoids loading PlayerActor assets for suits that the player doesn't have). If ~~anything else is~~ missing from the list, they ~~will~~ still be loaded, ~~but this step will cause~~ the game to hang ~~until~~ the ~~load~~ is ~~finished~~.		* Finally the game loads assets being used by the new area that are missing from the list. This primarily means [[SCAN (File Format)\|SCAN]] files (in MP1), the skybox model, and assets being used by PlayerActor animsets (this avoids loading PlayerActor assets for suits that the player doesn't have). If any other assets were missing from the list, they are loaded here. It's very important to know that contrary to the previous steps, assets that still aren't in memory at this point are loaded ''synchronously'', which means the game will hang until loading is complete; therefore, you want as few assets as possible to be left to load at this point. (You can observe this hang even in the base game - there is a small freeze towards the end of loading Artifact Temple.)
	* Note that any assets that are already in memory will be skipped.		* Note that any assets that are already in memory will be skipped.

>Aruki: /* Optimizing Load Times */

2018-06-14T19:37:50Z

Optimizing Load Times

← Older revision		Revision as of 19:37, 14 June 2018
Line 124:		Line 124:
	When rebuilding new paks, it's extremely important that asset order be optimized in order to minimize the distance the game has to seek on the disc in order to load any given asset. A poorly optimized pak can easily have 30+ second load times on small rooms. There are three main things that are done in order to keep load times fast.		When rebuilding new paks, it's extremely important that asset order be optimized in order to minimize the distance the game has to seek on the disc in order to load any given asset. A poorly optimized pak can easily have 30+ second load times on small rooms. There are three main things that are done in order to keep load times fast.

	* In world paks, assets are often duplicated so they appear multiple times within the pak; if an asset is used by a lot of different areas this can help reduce seek distance, but has the downside of bloating the total file size of the pak. To ensure the best balance between the two, asset duplication was ~~likely~~ flagged on a per-area basis, as ~~large~~ areas tend to have a ~~lot of duplicate resources~~, ~~while smaller ones that load quickly anyway don~~'t.		* In world paks, assets are often duplicated so they appear multiple times within the pak; if an asset is used by a lot of different areas this can help reduce seek distance, but has the downside of bloating the total file size of the pak. To ensure the best balance between the two, asset duplication was flagged on a per-area basis, as larger areas take longer to load and would benefit from the faster load speed, whereas small rooms tend to load quickly anyway even without duplicates. The flag that was used for this only existed in Retro's raw files and isn't present in the final game's cooked data. You can recreate it by analyzing the pak and noting down which rooms have duplicate assets (bearing in mind that the assets used by a room always precede that room's MREA asset). If you are working strictly from a pre-unpacked pak, you can't replicate the flag.
	* Assets are generally grouped by load order - things that are used together are adjacent in the pak, and an asset's dependencies typically appear immediately preceding the asset itself. This grouping helps ensure that even when assets aren't duplicated the game will still be able to do just one large seek and then load a bunch of assets at once, instead of having to do a separate seek for each asset.		* Assets are generally grouped by load order - things that are used together are adjacent in the pak, and an asset's dependencies typically appear immediately preceding the asset itself. This grouping helps ensure that even when assets aren't duplicated the game will still be able to do just one large seek and then load a bunch of assets at once, instead of having to do a separate seek for each asset.
	* Once you have assets stored in the pak in the right order, you need to make sure the game actually reads them in that order! The exact order that assets are loaded in is specified by the dependency list in the [[MLVL (File Format)\|MLVL]] file; as such, the MLVL list should also have assets grouped in the same way and the same order and should at least roughly match the order files are positioned in the pak.		* Once you have assets stored in the pak in the right order, you need to make sure the game actually reads them in that order! The exact order that assets are loaded in is specified by the dependency list in the [[MLVL (File Format)\|MLVL]] file; as such, the MLVL list should also have assets grouped in the same way and the same order and should at least roughly match the order files are positioned in the pak.

>Aruki at 06:13, 28 March 2017

2017-03-28T06:13:16Z

← Older revision		Revision as of 06:13, 28 March 2017
Line 138:		Line 138:
	== Tools ==		== Tools ==

	* [https://drive.google.com/file/d/0B9MLV21H7SDvcGc1QjRfSTFJNk0/view?usp=sharing PakTool] by ~~Parax~~		* [https://drive.google.com/file/d/0B9MLV21H7SDvcGc1QjRfSTFJNk0/view?usp=sharing PakTool] by Aruki

	[[Category:File Formats]]		[[Category:File Formats]]
	[[Category:Metroid Prime]]		[[Category:Metroid Prime]]
	[[Category:Metroid Prime 2: Echoes]]		[[Category:Metroid Prime 2: Echoes]]

>Aruki: /* Optimizing Load Times */

2016-08-19T10:10:48Z

Optimizing Load Times

← Older revision		Revision as of 10:10, 19 August 2016
Line 122:		Line 122:
	== Optimizing Load Times ==		== Optimizing Load Times ==

	When rebuilding new paks, it's extremely important that asset order be optimized in order to minimize the distance the game has to seek on the disc in order to load any given asset. A poorly optimized pak can easily have 30+ second load times on small rooms. There are ~~two~~ main things that are done in order to keep load times fast.		When rebuilding new paks, it's extremely important that asset order be optimized in order to minimize the distance the game has to seek on the disc in order to load any given asset. A poorly optimized pak can easily have 30+ second load times on small rooms. There are three main things that are done in order to keep load times fast.

	* In world paks, assets are often duplicated so they appear multiple times within the pak; if an asset is used by a lot of different areas this can help reduce seek distance, but has the downside of bloating the total file size of the pak. To ensure the best balance between the two, asset duplication was likely flagged on a per-area basis, as large areas tend to have a lot of duplicate resources, while smaller ones that load quickly anyway don't.		* In world paks, assets are often duplicated so they appear multiple times within the pak; if an asset is used by a lot of different areas this can help reduce seek distance, but has the downside of bloating the total file size of the pak. To ensure the best balance between the two, asset duplication was likely flagged on a per-area basis, as large areas tend to have a lot of duplicate resources, while smaller ones that load quickly anyway don't.
	* Assets are generally grouped by load order - things that are used together are adjacent in the pak, and an asset's dependencies typically appear immediately preceding the asset itself. This grouping helps ensure that even when assets aren't duplicated the game will still be able to do just one large seek and then load a bunch of assets at once, instead of having to do a separate seek for each asset.		* Assets are generally grouped by load order - things that are used together are adjacent in the pak, and an asset's dependencies typically appear immediately preceding the asset itself. This grouping helps ensure that even when assets aren't duplicated the game will still be able to do just one large seek and then load a bunch of assets at once, instead of having to do a separate seek for each asset.
			* Once you have assets stored in the pak in the right order, you need to make sure the game actually reads them in that order! The exact order that assets are loaded in is specified by the dependency list in the [[MLVL (File Format)\|MLVL]] file; as such, the MLVL list should also have assets grouped in the same way and the same order and should at least roughly match the order files are positioned in the pak.

	For every asset loaded, the game checks every copy of the asset in the pak and loads whichever duplicate is the shortest distance from the last read position (the end of the last asset that was loaded). When loading an area, this is the order the game loads assets in:		For every asset loaded, the game checks every copy of the asset in the pak and loads whichever duplicate is the shortest distance from the last read position (the end of the last asset that was loaded). When loading an area, this is the order the game loads assets in:
	* First the game goes down the area dependency list in the [[MLVL (File Format)\|MLVL]] file and loads every file in the list for every active layer. The game follows the exact order specified by the list.		* First the game goes down the area dependency list in the [[MLVL (File Format)\|MLVL]] file and loads every file in the list for every active layer. The game follows the exact order specified by the list.
	* After loading all area dependencies, the area itself is loaded.		* After loading all area dependencies, the area itself is loaded.
	* Finally the game loads assets being used by the new area that are missing from the list. This primarily means [[SCAN (File Format)\|SCAN]] files, the skybox model, and assets being used by PlayerActor animsets (this avoids loading PlayerActor assets for suits that the player doesn't have). If anything else is missing from the list, they will still be loaded, but this step will cause the game to hang until the load is finished.		* Finally the game loads assets being used by the new area that are missing from the list. This primarily means [[SCAN (File Format)\|SCAN]] files (in MP1), the skybox model, and assets being used by PlayerActor animsets (this avoids loading PlayerActor assets for suits that the player doesn't have). If anything else is missing from the list, they will still be loaded, but this step will cause the game to hang until the load is finished.
	* Note that any assets that are already in memory will be skipped.		* Note that any assets that are already in memory will be skipped.

>Aruki at 10:00, 10 August 2016

2016-08-10T10:00:46Z

← Older revision		Revision as of 10:00, 10 August 2016
Line 4:		Line 4:

	Note that the Metroid Prime 3 E3 prototype uses this pak format as well, but has 64-bit file IDs. This is the only difference; the version number is still the same, so there isn't an easy way to check for this variation of the format.		Note that the Metroid Prime 3 E3 prototype uses this pak format as well, but has 64-bit file IDs. This is the only difference; the version number is still the same, so there isn't an easy way to check for this variation of the format.

	~~{{research\|minor\|Some research is needed to figure out the optimal way to order files to optimize load times.}}~~

	__TOC__		__TOC__