VFS Error while trying to read a file (JavaScript)

ramtzok1 · June 20, 2019

EDIT: The repository https://github.com/ram1660/0ad

Hey guys, long time has passed,and I hoped I could resurface with a similar question to previous ones:
Me and my partner have continued working on our ML AI and on the brink of finishing, but one error is continuously bothering us: When reading the answer.txt file that contains the % of winning an engagement using Engine.ReadFile() in the assignDefenders function , we usually see an error in game that reads: " CVFSFile: file simulation/ai/petraML/mldata/answer.txt could not be opened(vfs_load: -110101)"

later on, if it runs too much, it usually crashes with the following tracing:

Assertion failed: "hFile != INVALID_HANDLE_VALUE"
Location: wfilesystem.cpp:285 (wtruncate)

Call stack:

wtruncate (wfilesystem.cpp:285)
    pathname = 0x00F8E0F4 -> 
        path = (unsupported basic_string<wchar_t,char_traits<wchar_t> >)
        separator = [8] { 92 ('\'), 104 ('h'), 413, 57282, 57788, 248, 3176, 104 ('h') }

    length = 32327 (0x0000000000007E47)

io::Store<io::DefaultCompletedHook,io::DefaultIssueHook> (io.h:315)
    pathname = 0x00F8E0F4 (see above)
    data = 0x39FDF000
    size = 32327 (0x00007E47)
    p = 0x00F8E0D8 -> 
        alignment = 1 (0x0000000000000001)
        blockSize = 0 (0x00000000)
        queueDepth = 1 (0x00000001)

    completedHook = 0x00F8E0F3 -> (io::DefaultCompletedHook)
    issueHook = 0x00F8E0F3 (see above)
    op = 
        m_FileDescriptor = 6 (0x00000006)
        m_OpenFlag = 7 (0x00000007)
        m_Offset = 0 (0x0000000000000000)
        m_Size = 32327 (0x0000000000007E47)
        m_Buffer = 0x39FDF000 (see above)

    file = 
        m_PathName = 
            path = (unsupported basic_string<wchar_t,char_traits<wchar_t> >)
            separator = 92 ('\')

        m_FileDescriptor = 4294967295 (0xFFFFFFFF)
        m_OpenFlag = 1 (0x00000001)


RealDirectory::Store (real_directory.cpp:57)
    this = (unavailable)
    name = 0x00F8E19C -> 
        path = (unsupported basic_string<wchar_t,char_traits<wchar_t> >)
        separator = "/]ŵ�øg"

    fileContents = 0x00F8E200 -> (shared_ptr<unsigned char>)
    size = 32327 (0x00007E47)

VFS::CreateFile (vfs.cpp:145)
    this = (unavailable)
    pathname = 0x00F8E218 -> 
        path = (unsupported basic_string<wchar_t,char_traits<wchar_t> >)
        separator = [8] { 47 ('/'), 248, 62208, 14840, 57988, 248, 43 ('+'), 0 }

    fileContents = 0x00F8E200 (see above)
    size = 32327 (0x00007E47)
    s = (`anonymous-namespace'::ScopedLock)
    file = 
        m_name = 
            path = (unsupported basic_string<wchar_t,char_traits<wchar_t> >)
            separator = 16

        m_size = 875861008 (0x34349410)
        m_mtime = 26288751805784504 (0x005D657B00F8E1B8)
        m_priority = 1 (0x00000001)
        m_loader = (shared_ptr<IFileLoader>)

    name = 
        path = (unsupported basic_string<wchar_t,char_traits<wchar_t> >)
        separator = 47 ('/')

    directory = 0x0A0BC4BC -> 
        m_files = (unsupported map<Path,VfsFile >)
        m_subdirectories = (unsupported map<Path,VfsDirectory >)
        m_realDirectory = (shared_ptr<RealDirectory>)
        m_shouldPopulate = 0 (0x00000000)


JSI_VFS::WriteJSONFile (jsinterface_vfs.cpp:223)
    pCxPrivate = 0x1C545740 -> { pScriptInterface = 0x1C545740 (see above), pCBData = 0x17979FE0 }
    filePath = 0x00F8E284 -> (unsupported basic_string<wchar_t,char_traits<wchar_t> >)
    val1 = 
        { (js::ValueOperations<JS::Handle<JS::Value> >) }
        ptr = 0x00F8E320 -> 
            data = 
                asBits = -515287411008 (0xFFFFFF88067A16C0)
                s = { payload = 
                        i32 = 108664512 (0x067A16C0)
                        u32 = 108664512 (0x067A16C0)
                        boo = 108664512 (0x067A16C0)
                        str = 0x067A16C0 -> (JSString)
                        sym = 0x067A16C0 (see above)
                        obj = 0x067A16C0 (see above)
                        cell = 0x067A16C0 (see above)
                        ptr = 0x067A16C0 (see above)
                        why = 108664512
                        word = 108664512 (0x067A16C0)
                        uintptr = 108664512 (0x067A16C0)
, tag = -120 }
                asDouble = -nan (0xFFFFFF88067A16C0)
                asPtr = 0x067A16C0 (see above)



    str = (unsupported basic_string<char,char_traits<char> >)
    path = 
        path = (unsupported basic_string<wchar_t,char_traits<wchar_t> >)
        separator = 47 ('/')

    rq = 
        mContext = 0x1AD21E18 -> (JSContext)

    val = 
        { 
                (js::ValueOperations<JS::Rooted<JS::Value> >)
 }
        stack = 0x1AD21E4C -> 0x00F8E208 -> 
            (js::RootedBase<void *>)
            stack = 0x1AD21E4C (see above)
            prev = 0x00F8E29C -> 
                (js::RootedBase<void *>)
                stack = 0x1AD21E4C (see above)
                prev = 0x00F8E810 -> 
                    (js::RootedBase<void *>)
                    stack = 0x1AD21E4C (see above)
                    prev = 0x00F8EBA4 -> 
                        (js::RootedBase<void *>)
                        stack = 0x1AD21E4C (see above)
                        prev = 0x00F8EC2C -> 
                            (js::RootedBase<void *>)
                            stack = 0x1AD21E4C (see above)
                            prev = 0x00F8ECE4 -> 
                                (js::RootedBase<void *>)
                                stack = 0x1AD21E4C (see above)
                                prev = 0x00000000
                                ptr = 0x06786A40

                            ptr = 0x00000000

                        ptr = 0x070D1220

                    ptr = 0x00000003

                ptr = 0x00000000

            ptr = 0x067A16C0 (see above)

        prev = 0x00F8E29C (see above)
        ptr = 
            data = 
                asBits = -515287411008 (0xFFFFFF88067A16C0)
                s = { payload = 
                        i32 = 108664512 (0x067A16C0)
                        u32 = 108664512 (0x067A16C0)
                        boo = 108664512 (0x067A16C0)
                        str = 0x067A16C0 (see above)
                        sym = 0x067A16C0 (see above)
                        obj = 0x067A16C0 (see above)
                        cell = 0x067A16C0 (see above)
                        ptr = 0x067A16C0 (see above)
                        why = 108664512
                        word = 108664512 (0x067A16C0)
                        uintptr = 108664512 (0x067A16C0)
, tag = -120 }
                asDouble = -nan (0xFFFFFF88067A16C0)
                asPtr = 0x067A16C0 (see above)



    buf = 
        m_capacity = 32768 (0x00008000)
        m_data = (shared_ptr<unsigned char>)
        m_size = 32327 (0x00007E47)


ScriptInterface::call<void,std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >,JS::Handle<JS::Value>,&JSI_VFS::WriteJSONFile> (nativewrapperdefns.h:125)
    cx = 0x1AD21E18 (see above)
    argc = 2 (0x00000002)
    vp = 0x00F8E308 -> 
        data = 
            asBits = -514943345344 (0xFFFFFF881AFC1D40)
            s = { payload = 
                    i32 = 452730176 (0x1AFC1D40)
                    u32 = 452730176 (0x1AFC1D40)
                    boo = 452730176 (0x1AFC1D40)
                    str = 0x1AFC1D40 -> (JSString)
                    sym = 0x1AFC1D40 (see above)
                    obj = 0x1AFC1D40 (see above)
                    cell = 0x1AFC1D40 (see above)
                    ptr = 0x1AFC1D40 (see above)
                    why = 452730176
                    word = 452730176 (0x1AFC1D40)
                    uintptr = 452730176 (0x1AFC1D40)
, tag = -120 }
            asDouble = -nan (0xFFFFFF881AFC1D40)
            asPtr = 0x1AFC1D40 (see above)


    a0 = (unsupported basic_string<wchar_t,char_traits<wchar_t> >)
    rq = 
        mContext = 0x1AD21E18 (see above)

    rval = 
        { 
                (js::ValueOperations<JS::Rooted<JS::Value> >)
 }
        stack = 0x1AD21E4C (see above)
        prev = 0x00F8E810 (see above)
        ptr = 
            data = 
                asBits = -541165879296 (0xFFFFFF8200000000)
                s = { payload = 
                        i32 = 0 (0x00000000)
                        u32 = 0 (0x00000000)
                        boo = 0 (0x00000000)
                        str = 0x00000000
                        sym = 0x00000000
                        obj = 0x00000000
                        cell = 0x00000000
                        ptr = 0x00000000
                        why = JS_ELEMENTS_HOLE
                        word = 0 (0x00000000)
                        uintptr = 0 (0x00000000)
, tag = -126 }
                asDouble = -nan (0xFFFFFF8200000000)
                asPtr = 0x00000000



    typeConvRet0 = true 

2C4CECD9

373971E8

js::jit::DoIteratorNewFallback (baselineic.cpp:11066)
    cx = (unavailable)
    frame = (unavailable)
    stub = (unavailable)
    value = (unavailable)
    res = (unavailable)


errno = 0 (No error reported here)
OS error = 32 (The process cannot access the file because it is being used by another process.)

It might not mean much, but after trying to figure out what causes this problem(running a ton of tests and creating a ton of simple,non-working functions in the engine scripts) we reached somewhat of a dead-end. It would mean so much if you guys could help us again regarding this issue.

Picture attached for reference. If you have any other questions you could review my previous threads and ask me directly.

We are going to present our work in Monday, and any help towards solving is deeply appreciated!

Edited June 22, 2019 by ramtzok1

**Stan`** · June 20, 2019

What os are you ?

Windows handles multiple file ownership badly.

Do you have a set up guide so I can try to reproduce it locally

ramtzok1 · June 20, 2019

I am on Windows. I can give you the files we modified + the script or share our repo if you would like!

**elexis** · June 20, 2019

1 hour ago, ramtzok1 said:

OS error = 32 (The process cannot access the file because it is being used by another process.)

Last line of the log. So gotta find some mechanism to prevent two processes from reading and writing simultaneously to the same file (https://en.wikipedia.org/wiki/Mutual_exclusion#Types_of_mutual_exclusion_devices )

**Stan`** · June 20, 2019

Yeah so as elexis said you need to find a way to put a semaphore on the file (or handle the case where it's used by another process)

On a side note, if you are not the one making this PR https://github.com/0ad/0ad/pull/25 you can use that to send commands directly to the engine. You'll probably have to figure some stuff out first, maybe contact the person

June 20, 2019

Trying to deal with this in Python code is easier than trying to do the same 0AD side.

ramtzok1 · June 20, 2019

1 hour ago, Stan` said:

What os are you ?

Windows handles multiple file ownership badly.

Do you have a set up guide so I can try to reproduce it locally

Me and my partner modified functions and code I think it won't be possible to reproduce without uploading the whole project to GitHub and getting Anaconda set up correctly on your side.

I will upload the project tomorrow.

ramtzok1 · June 20, 2019

15 minutes ago, (-_-) said:

Trying to deal with this in Python code is easier than trying to do the same 0AD side.

42 minutes ago, elexis said:

Last line of the log. So gotta find some mechanism to prevent two processes from reading and writing simultaneously to the same file (https://en.wikipedia.org/wiki/Mutual_exclusion#Types_of_mutual_exclusion_devices )

We dealt with that problem where there was a data race between Python and the game where they both tried to access the same file. We added exceptions handling on the Python side and making sure no other process is using the file.

I don't think the Python script is related to that problem any more.

As I said I will upload the whole project including the ML model so you will be able to see every step we are making and maybe spot where we did wrong.

Edited June 20, 2019 by ramtzok1
Forgot to quote elexis

**Stan`** · June 20, 2019

5 minutes ago, ramtzok1 said:

Me and my partner modified functions and code I think it won't be possible to reproduce without uploading the whole project to GitHub and getting Anaconda set up correctly on your side.

I will upload the project tomorrow.

Good, I will try to find some time, no promises though.

**elexis** · June 21, 2019

Wasn't the file truncated to zero bytes and then deleted or something? Like that being two steps that can be affected by concurrency, not only one?

4 hours ago, ramtzok1 said:

We dealt with that problem where there was a data race between Python and the game where they both tried to access the same file. We added exceptions handling on the Python side and making sure no other process is using the file.

I don't think the Python script is related to that problem any more.

It does speak of a different process though. And once python opens file access (regardless of exceptions), 0ad can't open that file for that time, no? It sounds like you dealt with the problem where python tries to open it when 0ad has it open already, but not the other way around.

ramtzok1 · June 21, 2019

15 hours ago, elexis said:

Wasn't the file truncated to zero bytes and then deleted or something? Like that being two steps that can be affected by concurrency, not only one?

It does speak of a different process though. And once python opens file access (regardless of exceptions), 0ad can't open that file for that time, no? It sounds like you dealt with the problem where python tries to open it when 0ad has it open already, but not the other way around.

What you say may be true, but unfortunately we tried to figure out to be sure and it didn't quite work yet.

Here is how we handle the file accessing, JS-side (defenseManger.js):

	let i = 0;
	while(i < 100000) // delaying the JavaScript Side so python can catch-up. 
		i++;
	if( Engine.isFileInUse("D:\\Users\\micha\\Desktop\\ram-and-michael\\binaries\\data\\mods\\public\\simulation\\ai\\petraML\\mlData\\answer.txt") && !Engine.FileExists("simulation/ai/petraML/mlData/answer.txt"))
		return -1;
	Engine.AppendToBuffer("a");
	Engine.WriteToFile("simulation/ai/petraML/mlData/zxccxz.txt"); // dummy file to check if worked. 
	let chance = undefined;
	
	do {

		warn("before reading ")
		chance = parseFloat(Engine.ReadFile("simulation/ai/petraML/mlData/answer.txt")); // place where error occurs. 
		this.counter += 1;
		warn("COUNTER: " + this.counter.toString())
		warn("READ CHANCE: " + chance.toString())

		
	} while (chance === undefined);
		
	
	
	
	Engine.AppendToBuffer("a");
	
	Engine.WriteToFile("simulation/ai/petraML/mlData/gameState.txt"); // for python to check if JS is done.

	return chance;

IsFileInUse() is a function that we created to check if a text file is in use, the code:

inline bool JSI_VFS::isFileInUse(ScriptInterface::CxPrivate* pCxPrivate, const std::wstring& filePath)
{
	struct stat buffer;
	return (stat((char*)filePath.c_str(), &buffer) == 0);
}

The only time we access Answer.txt in the python side is to write the output from our model. The only problem I can figure out regrading the one-sided solution is the fact that JS is a lot faster and just somehow manages to try and check if Answer.txt exists and in use before + reading it before it is even created.

NOTE: we also tried to use the ReadFile function on a dummy file that we created to see if reading just a regular file that is not accessed brings trouble, and we faced the same error code you see regarding the file reading in the main post.

**Stan`** · June 21, 2019

7 minutes ago, ramtzok1 said:
What you say may be true, but unfortunately we tried to figure out to be sure and it didn't quite work yet.

Here is how we handle the file accessing, JS-side (defenseManger.js):
	let i = 0;
	while(i < 100000) // delaying the JavaScript Side so python can catch-up. 
		i++;
	if( Engine.isFileInUse("D:\\Users\\micha\\Desktop\\ram-and-michael\\binaries\\data\\mods\\public\\simulation\\ai\\petraML\\mlData\\answer.txt") && !Engine.FileExists("simulation/ai/petraML/mlData/answer.txt"))
		return -1;
	Engine.AppendToBuffer("a");
	Engine.WriteToFile("simulation/ai/petraML/mlData/zxccxz.txt"); // dummy file to check if worked. 
	let chance = undefined;
	
	do {

		warn("before reading ")
		chance = parseFloat(Engine.ReadFile("simulation/ai/petraML/mlData/answer.txt")); // place where error occurs. 
		this.counter += 1;
		warn("COUNTER: " + this.counter.toString())
		warn("READ CHANCE: " + chance.toString())

		
	} while (chance === undefined);
		
	
	
	
	Engine.AppendToBuffer("a");
	
	Engine.WriteToFile("simulation/ai/petraML/mlData/gameState.txt"); // for python to check if JS is done.

	return chance; 
IsFileInUse() is a function that we created to check if a text file is in use, the code:
inline bool JSI_VFS::isFileInUse(ScriptInterface::CxPrivate* pCxPrivate, const std::wstring& filePath)
{
	struct stat buffer;
	return (stat((char*)filePath.c_str(), &buffer) == 0);
}
The only time we access Answer.txt in the python side is to write the output from our model. The only problem I can figure out regrading the one-sided solution is the fact that JS is a lot faster and just somehow manages to try and check if Answer.txt exists and in use before + reading it before it is even created.

NOTE: we also tried to use the ReadFile function on a dummy file that we created to see if reading just a regular file that is not accessed brings trouble, and we faced the same error code you see regarding the file reading in the main post.

I believe the issue is the file being hotloaded like the js scripts. I'm not sure exactly which folder is being hotloaded but moving it out of there and loading it only on demand might work.

**elexis** · June 21, 2019

Another common trick to avoid issues like that is to rename the file before doing something with it, so that the other process doesn't get into the opportunity of doing something to it.

ramtzok1 · June 21, 2019

1 hour ago, Stan` said:

I believe the issue is the file being hotloaded like the js scripts. I'm not sure exactly which folder is being hotloaded but moving it out of there and loading it only on demand might work.

Can you please explain what does hotloaded mean? I'm unfamiliar with the term.

1 hour ago, elexis said:

Another common trick to avoid issues like that is to rename the file before doing something with it, so that the other process doesn't get into the opportunity of doing something to it.

Interesting idea! Is there a good way to rename a file in JS?

EDIT: I gave it some thought and I can't really see how it fits our code, since our objective is that the Python script will create a file and that file will be processed by the JavaScript.

Edited June 21, 2019 by ramtzok1

**Stan`** · June 21, 2019

5 hours ago, ramtzok1 said:

Can you please explain what does hotloaded mean? I'm unfamiliar with the term.

The game periodically checks for file changes and reload them if they changed. This allows you to change code in a script while the game is running for instance.

ramtzok1 · June 22, 2019

11 hours ago, Stan` said:

The game periodically checks for file changes and reload them if they changed. This allows you to change code in a script while the game is running for instance.

I see. I guess it one of the main folders of the game, and not our custom-made,right?
also, I did want to move the actual files to a place like My Documents, because I suspected a conflict early on, but as far as I understood the Engine.ReadFile and WriteFile wouldn't work with it, right? (The path needs to be like the one we used in the code examples above,and not absolute?)

Also, is there a place where I can check error codes? I tried to change the way we handle a time when JS tries to access the .txt, and got the following error code:

Assertion failed: "el_size != 0"
Location: wdbg_sym.cpp:848 (dump_sym_array)

Call stack:

While generating an error report, we encountered a second problem. Please be sure to report both this and the subsequent error messages.
errno = 2 (Error during IO)
OS error = 0 (no error code was set)

The changed code:

	winningPetential = this.getMLData();
		if(winningPetential == -1) // Basically, the file reading was unsuccessful. 
		{
			aMin -= 1;
			continue
		}

**Stan`** · June 22, 2019

15 minutes ago, ramtzok1 said:

I see. I guess it one of the main folders of the game, and not our custom-made,right?
also, I did want to move the actual files to a place like My Documents, because I suspected a conflict early on, but as far as I understood the Engine.ReadFile and WriteFile wouldn't work with it, right? (The path needs to be like the one we used in the code examples above,and not absolute?)

I guess you could try an absolute path. One could also try in a mod in my documents there is a 0ad folder in my games

16 minutes ago, ramtzok1 said:

Also, is there a place where I can check error codes? I tried to change the way we handle a time when JS tries to access the .txt, and got the following error code:

I guess those are generic error codes though you could check what calls the function at wdbg_sym.cpp line 848

ramtzok1 · June 22, 2019

2 minutes ago, Stan` said:

I guess you could try an absolute path. One could also try in a mod in my documents there is a 0ad folder in my games

I guess those are generic error codes though you could check what calls the function at wdbg_sym.cpp line 848

Okay, should I try just using the absolute path into a folder i create in the mod folder in My games?

I'll also chekc the cpp file, thanks!

**Stan`** · June 22, 2019

9 minutes ago, ramtzok1 said:

Okay, should I try just using the absolute path into a folder i create in the mod folder in My games?

Yeah, Or anywhere else really I never tried it before

ramtzok1 · June 22, 2019

Well, I tried to create a dummy file and just see if I could read data from it, but apparently it is not working at all, regardless of using a file or anything else. Does anyone know what could be the reason for that? I'm pretty sure that is what happens too since the VFS_load (Whatever that is) is the same that was for us beforehand.

Edited June 22, 2019 by ramtzok1

**Stan`** · June 22, 2019

25 minutes ago, ramtzok1 said:

Well, I tried to create a dummy file and just see if I could read data from it, but apparently it is not working at all, regardless of using a file or anything else. Does anyone know what could be the reason for that? I'm pretty sure that is what happens too since the VFS_load (Whatever that is) is the same that was for us beforehand.

Try a path without spaces (Just in case)

ramtzok1 · June 22, 2019

13 minutes ago, Stan` said:

Try a path without spaces (Just in case)

Yeah,It doesn't work. I guess it just defaults to everything in the simulation folder.

**Stan`** · June 22, 2019

Mmh look if you can access non vfs functions by creating an interface

Else try the simulation data folder iirc it's not hotloaded. Neither are the templates.

June 22, 2019

You might want to try another approach for IPC. (Inter-process communication).

What was wrong with the client-server approach?

ramtzok1 · June 22, 2019

Just now, (-_-) said:

You might want to try another approach for IPC. (Inter-process communication).

What was wrong with the client-server approach?

I tried to make an E-Net server-client, however I was confused about how to make the connection also my guide told me and to my partner to do not tarry on it if we are getting stuck and just stick to the idea of the files which made sense because it "worked" until we found all of these problems.

VFS Error while trying to read a file (JavaScript)

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Guest

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Guest

Link to comment

Share on other sites

Link to comment

Share on other sites

Join the conversation