intro
this document, and the series of posts derived from it will chronicle the trials and tribulations of getting this research project both back up and running, and also adding new features into it.
so what is going on here?
i was commissioned to do a bit of a science project… the project, which for anonymity i will call 'vt' is, in short, is to compare the performance of a selection of scripting languages in a series of standardized tests in the context of them being embedded in C++.
progress
[X]make it go[X]angelscript[-]luau[-]haxe (optional)[-]quickjs[X]test 1[X]test 2[ ]make it actually execute the code (failed)[ ]spend more time debugging to see why its even failing (failed)
[-]v8, spidermonkey, hermes[X]v8[X]test 1[X]test 2[X]test 3
[-]spidermonkey[X]implement test 1[X]implement test 2[X]implement test 3[ ]confirm that it all works
[-]hermes[ ]figure out how its classes work[X]hermes disqualified due to no public api
week 1
hereweek 2
hereweek 3
hereweek 4
day 1
so ive started off the day posting last week's update, which in typical fashion, found a way to fuck up because of course it did. i think its a cohost bug though so im not going to waste any time on it beyond adding a note to the post describing a workaround.
cmake package manager
now, time to look at that package manager and see if it will be of help here, or just a mess. its looking to me that this may primarily be designed for handling the downloading of repos, which is indeed quite useful for keeping things up to date but might not be as helpful for things already downloaded. agh, tragedy struck!
Dependent on good CMakeLists: Many libraries do not have CMakeLists that work well for subprojects. Luckily this is slowly changing, however, until then, some manual configuration may be required (see the snippets below for examples). For best practices on preparing projects for CPM, see the wiki.
- CPM README.md
well. damn.
i'll have to keep this in mind for future projects though! especially seeing as it makes adding both benchmark and boost trivial.
creating library projects
ok so i need to create a library project for these guys, lets start with:
- spidermonkey
im chosing spidermonkey as its been by far the most promising in actually getting going. now, lets see what i actually have to do to make a library project. while i was looking for information, i also had a stupid but potentially genius idea as to how to get a non prebuilt spidermonkey. i might be able to get a list of all .h files and then scan the actual source tree for matching .cpp files. i ought to be able to whip up a script that can do this for all 500 some odd files.
anyway. back to looking for info.. ah, remember when stack overflow used to be useful?
so based on this, i think i need to declare a project, execute addlibrary() and then slap a boatload of source files in there. after doing that i need to declare jsapi.h as the public api, then create something called an "install rule"
ive managed to do most of that, but its becoming clear im going to actually have to do that thing to get the sources
so to do that i will need this script to:
find -ipath "*.h*" #over the headers we have extracted #after which i will need to put the results of that into a list #then i will need to strip paths only leaving the name find -ipath "*.cpp*" #over the files in the firefox source tree into another list. #then i will need to iterate over each, saving a new list containing those whos strings contain strings from the first list cp ${THIRD_LIST} ${VT_SOURCE_DIR}/deps/spidermonkey #il probably do it into a new temp directory in case this goes completely sideways #i also need to see if it will autocreate subdirectories for me...
ok, just one small snag, im not any good with bash scripts, so this might take a sec, especially that string stripping bit. unless i can do all this in lua instead… well.. cracks knuckles lets just go for it… ok yeah i have no clue how to do the more complex list processing part of this from bash, so i will just call a lua script that can handle that part more ergonomically
ok so after plenty of foxing around, i managed to get it to get me all of the source files. unfortunately, it completely enilated the tree structure they were once in. but hey, at least i actually have the files now, all way too many of them
no the script must have fucked up somehow, theres nearly 2k source files when there was only 500 something headers…
maybe i can just go head in the sand and pretend there is no problem at all here and just use this as a global sources directory?
the ongoing issues of yesterday spill into today.
i decided to put the source files in with the headers. this doesnt really make sense from a folder naming standpoint, but all of the source files refer to headers paths that only make sense when viwed from a perspective of already being in that folder.
much to my shock, running cmake . on the new file i made resulted in no errors. im going to see if i cant include this directory now…
ok im thinking those extra sources might be problematic. so i did some command line magic to remove some offending files… specifically
rg -F --files-with-matches "StdAfx" | xargs rmStdAfx.h being a file defined seversal times in the original source tree, and ultimately not even being necessary here anyway
well, now im presented with a new problem.
cmake --build . -t test-spidermonkey [ 0%] Built target benchmark [ 0%] Built target benchmark_main deps/spidermonkey/CMakeFiles/spidermonkey.dir/build.make:62: CMakeFiles/spidermonkey.dir/depend.make: No such file or directory make[3]: *** No rule to make target 'CMakeFiles/spidermonkey.dir/depend.make'. Stop. make[2]: *** [CMakeFiles/Makefile2:1274: deps/spidermonkey/CMakeFiles/spidermonkey.dir/all] Error 2 make[1]: *** [CMakeFiles/Makefile2:584: CMakeFiles/test-spidermonkey.dir/rule] Error 2 make: *** [Makefile:309: test-spidermonkey] Error 2
im not even entirely sure what this means… no rule to make target 'depend.cmake' huh? il have to look up what that is. no results so far.. fun.
how naive of me to think that i could get more than one of these done in a day….
just great…
day 2
i was occupied for this day
day 3
i was occupied for this day
day 3
i was occupied for this day. additional days will be added to this week.
day 4
so i heard back from my client. hermes is officially dropped due to having no api. they also said that benchmark was originally included as a compiled lib. so i should go back and check how things were when i started to see if that will help getting spidermonkey going.
review of the old cmakelists
lets start by grabbing every line referencing benchmark oh, well first off, they did that with v8, not benchmark, because apparently i cant read.
with that out of the way, its not even cmakelists, but a .cmake file. ive never dealt with those yet
# the process behind getting and compiling a v8 binary is very complicated :( # I just copied the final artifacts into the deps/ directory add_library(libv8 STATIC IMPORTED) set_target_properties(libv8 PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/deps/v8/obj/libv8_monolith.a INTERFACE_INCLUDE_DIRECTORIES "${CMAKE_SOURCE_DIR}/deps/v8/include" ) target_link_libraries(libv8 INTERFACE pthread)
thats it, apparently. this looks very simple actually. however, given that the commit message is "v8 doesn't work :(" i expect a hitch incoming.
the attempt begins
so im going to start by restoring the backup i made of how all the headers were before i used my source extractor script. with that done, im going to look through the files and find the compiled binary. ah so i found what i think is the bin. doing cat on it makes my system freak out like you wouldnt believe, so i think its safe to say thats an executable file. doing cat|more on it tells me its an elf.
readelf bin/js115 -h ELF Header: Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: DYN (Position-Independent Executable file) Machine: Advanced Micro Devices X86-64 Version: 0x1 Entry point address: 0x16b130 Start of program headers: 64 (bytes into file) Start of section headers: 363079544 (bytes into file) Flags: 0x0 Size of this header: 64 (bytes) Size of program headers: 56 (bytes) Number of program headers: 14 Size of section headers: 64 (bytes) Number of section headers: 44 Section header string table index: 43
elfiness confirmed. how seasonal. (december at time of writing) according to wikipedia, dyn means it is a "shared object file"
there are other files in the lib/ folder. cat|more-ing them somehow managed to crash discord, btop, and freeze the tab it was running in. how? i dont know. 3x sigkill coming right up! in any case, we've got:
tree lib
lib
├── libjs_static.ajs
├── libmozjs-115.so
└── pkgconfig
└── mozjs-115.pc
2 directories, 3 files
im not fammilliar with any of these filtypes. i can find no information whatsoever on what a .ajs file is other than vaiguely being "binary data", thanks. i figured that out on my own. a .so file is apparently a 'shared object'. which does appear to be a c/cpp thing. a .pc file is apparently a thing for the linker.
the .cmake file above links v8 in as static, i wonder if i can do that but have it as shared instead… yeah, you absolutely can just sorta do that. the stuff i found online is literally just a 2 liner so il see if i cant just slap it directly into CMakeLists.txt and bam, im bombarded with the same massive wall of errors as before, indicitave of a library not being found. i tried moving it into its own .cmake file and including that in the same manner as v8. no change. out of curiosity, i tried executing bin/js115, and it is a cli js repl i assume this means it is an executable, rather than a library, but i might try using it anyway since the .so file didnt work. no change. i realised that i didnt include the includes directory like i did for v8. however, i cannot.
CMake Error at CMakeLists.txt:163 (target_include_directories): Cannot specify include directories for target "test-spidermonkey" which is not built by this project.
well thats one hell of a catch-22 isnt it. i cant build the test without the library, i cant build the library because its precompiled, but i have to build the library because cmake says so apparently. i went ahead and globbed the headers again to see if that would help. it did nothing. its like its not even trying to include the header at all. nothing i do is working.
i went ahead and commented out the entirety of spidermonkey-benchmarks.cpp except for the include. lvim instantly freaked out and is now frozen. apparently it is not a fan of good ol * *
ive continued poking around for a while and found nothing. i managed to crash a lot of things again by experimentally just including one of the compiled files. maybe this article i found will help? oh hey, that .pc file straight up says that its -lmozjs-115 renaming everything in cmake to mozjs-115 didnt fix a damn thing, but i did find out that part of the linking is technically happening correctly. so ive deduced its finding the header file correctly. it is not however, actually linking the damn library. why?
it reccomended i use something called pkgconfig, which automagically reads the .pc file and sets everything up. sounds cool enough and hey we got a new error!
{path to vt redacted}/js/spidermonkey-benchmarks.cpp:1: /usr/include/mozjs-115/js-config.h:31:4: error: #error "SpiderMonkey was configured with --disable-debug, so DEBUG must be not defined when including this header"
odd that it is using the system installed version of spidermonkey. but on the other hand, this means it actually found spidermonkey! at this rate i dont even care that its using the system one. i can just recompile it to not be –disable-debug let the recompilation begin! the recompilation failed to overwrite the existing install it seems im going to have to figure out how to manually remove this.
day 5
monke
so instead of doing that, im just going to see if ic ant do a release build. lets see if that works no. of course it doesnt same issue as earlier where it just doesnt i actually have no idea where to go from here. its just the same problem again and again, it should be linking, but no matter what i do it just doesnt weirdly, looking through jsapi.h, clangd cant make sense of half of it, its like something about the file is completely scrambling it. it cant find one file that is clearly there, it can find another file FROM THE SAME FOLDER declared in the exact same way! that is there, it labels well over half the imports as unused. it doesnt understand the concept of "namespace" suddenly. its so bizzaire. are there some invisible characters messing up the header or anything? it doesnt make any sense.
day 6
spidermonkey
so yesterday ended in complete failure. nothing was achieved despite multiple tests. i still have absolutely no clue why the code wont compile. i think it makes the most sense if instead, i look back on whats up with quickjs oh, right. i had already written off quickjs as unusable… well what now then? perhaps there are other js engines to look at? ah no that is basically all the open source ones. according to wikipedia, the game '0 A.D.' uses spidermonkey, maybe poking around in its source files will reveal something? well i found their spidermonkey finally. it seems they are using version 93, which is much older than ours, version 115. https://trac.wildfiregames.com/browser/ps/trunk/libraries/source/spidermonkey as we can see, it is set up very differently compared to the version we have. i guess in the 20 versions in between they managed to completely restructure the library? unfortunately i have no clue how to dl anything off of this site so im not sure how i can actually poke around with it… i have no idea what trac even is, but judging from their logo, im going to assume its run by fellow furries. is this some kind of pre git era version control? oh, no it isnt? it claims to support git. maybe i can just clone..? that would be a no. ah worth a shot. well i found a download button for individual files. but it curiously is not there for folders, making that a nonstarter… ah, here we go, source download..
day 7
spidermonkey
continuing off of yesterday, time to extract that source file and see how things differ. well, i can indeed see that a few of the dependencies in this project are cmake based… and i can see that all the files that refer to jsapi.h exist within a folder called "scriptinterface" so im guessing this is where they are driving spidermonkey from. i think thats where i should focus my efforts.
rg -F --files-with-matches "jsapi.h"
source/scriptinterface/tests/test_ScriptConversions.h
source/scriptinterface/ScriptContext.cpp
source/scriptinterface/ScriptExtraHeaders.h
source/scriptinterface/ScriptTypes.h
of these, only one is a .cpp. seems like the best place to start. fantastic, my system cant recognise any of the symbols from this jsapi.h either. thats more to be expected since i just grabbed this project, but dissapointing. interestingly, it also refers to a "precompiled.h".. wonder what thats about. clangd says file not found but i bet its wrong.
find -ipath "*precompiled.h*"
./source/lib/precompiled.h
./source/collada/precompiled.h
./source/pch/gui/precompiled.h
./source/pch/test/precompiled.h
./source/pch/simulation2/precompiled.h
./source/pch/lobby/precompiled.h
./source/pch/atlas/precompiled.h
./source/pch/glooxwrapper/precompiled.h
./source/pch/lowlevel/precompiled.h
./source/pch/scriptinterface/precompiled.h
./source/pch/network/precompiled.h
./source/pch/graphics/precompiled.h
./source/pch/tinygettext/precompiled.h
./source/pch/engine/precompiled.h
./source/tools/atlas/AtlasUI/Misc/precompiled.h
oh.. there are several. the first one seems like its the general one for the project. hm. doesnt seem to be too much of interest there. the cpp file from earlier looked pretty normal too, maybe there were slight api changes but i didnt really notice any.
well um.. what about build files then. if the main project isnt cmake, is it straight make? there are several makefiles sprinkled about, but none seem to be for the main project itself. perhaps the website has more information then..?
well they do have build instructions, anything interesting in there> completely unrelated, but the build instructions do mention the concept of distributed compilation, which is really cool. il have to investigate that later. ah so yes, it does actually just run with make. i need to look for a folder they call '0ad/build/workspaces'
well well well… (update-workspaces.sh)
# Check for whitespace in absolute path; this will cause problems in the # SpiderMonkey build (https://bugzilla.mozilla.org/show_bug.cgi?id=459089) # and maybe elsewhere, so we just forbid it # Use perl as an alternative to readlink -f, which isn't available on BSD or OS X SCRIPTPATH=`perl -MCwd -e 'print Cwd::abs_path shift' "$0"` case "$SCRIPTPATH" in *\ * ) die "Absolute path contains whitespace, which will break the build - move the game to a path without spaces" ;; esac
this doesnt actually apply to me since i know better than to do development from paths with spaces after previous experiences, but its interesting nonetheless i found unequivocal evidence it just uses GNU make(the script straight up says so), but where is the makefile?
(update-workspaces.sh again)
# Now run premake to create the makefiles
something called "premake" specifically, it calles "premake5" so, what the heck is premake? remember when homepages used to have all the information you needed on them? at least i can tell instantly its a build system. must be a competitor to cmake. oh sick its just lua! well thats neat. the config files look like a simplified version of the cmake configs. but i guess its actually lua with extra keywords? …and a shocking lack of equals signs… or 'do', or 'end' well anyway this means im looking for a .lua file! oh theres a fuckton of .lua files. ill need to be more specific. update-workspaces.sh specifically calles premake5.lua of which,,
find -ipath "*premake5.lua*"
./build/premake/premake5.lua
./build/premake/premake5/premake5.lua
./build/premake/premake5/binmodules/example/premake5.lua
./build/premake/premake5/binmodules/luasocket/premake5.lua
./build/premake/premake5/contrib/mbedtls/premake5.lua
./build/premake/premake5/contrib/zlib/premake5.lua
./build/premake/premake5/contrib/curl/premake5.lua
./build/premake/premake5/contrib/lua/premake5.lua
./build/premake/premake5/contrib/luashim/premake5.lua
./build/premake/premake5/contrib/libzip/premake5.lua
./build/premake/premake5/tests/folder/premake5.lua
/build/premake/premake5.lua it is. just as i was beginning to worry that was a premade file for premake5, it references dependencies that 0ad uses… including spidermonkey. lets dig through and find all references to spidermonkey.. its only references are as a string in several tables called "externlibs" and one table called "usedexternlibs"
so that may or may not be a dead end…?
day 8
so yesterday seems to be a bit of a bust?
im back to having little to no leads on what to do here.
perhaps there are other areas of the codebase that need cleaning up? well to my surprise im not finding much.. i could possibly move the luau test into the lua folder, removing that entirely, but i dont see much need to and that may introduce additional bugs. im going to take another crack at quickjs using the debugger. maybe i can find out why on earth its null..? what in the hell? either the debugger is playing tricks on me, or a value becomes 0 in between two lines somehow…? somehow burried in there, ive encountered the possibility that it thinks it is "not a function"? which somehow gets lost in the sauce, and everything becomes 0. interesting. this is very confusing because i very clearly assert() that it is a function before calling it.
assert(JS_IsFunction(ctx,func));
not only that, but there is also an if statement before that doing basically the exact same thing from earlier in my debugging process… a fair bit of debugging later and i managed to crash the debugger, great. i found another debugger called seer, which is also a gdb front end, just as ddd is, but this one appears to have be far less graphically antequated. perhaps it will prove more functional as well? before i forget, and i will forget, it's invoked with 'seergdb' ok so i just realized something debugging shows me that part of the code says that it definitely a JSCLASSBYTECODEFUNCTION, but the function that runs it begs to differ. why cant the debugger find basicstring.h? isnt that just a normal file for the language? digging through again, somehow the name of the function gets lost somewhere, il have to keep digging… an interesting finding, my previous thought can be disregarded as that was from too early in the code to be related to the function being called what does happen though, is that
JSValue r = JS_Eval(ctx, script.c_str(), strlen(script.c_str()),"",0);
gives us, according to the debugger:
r:u = {int32 = 0, float64 = 0, ptr = 0x0}, tag = 3
a null pointer in the wild! look at that. which means it fails to actually evaluate the source code at all! why did this not come up in previous tests? why does the later functions not even catch this error state? all this happens before we even try to call anything, so it ought to be affecting the result of BMLoadNBody as well! testing confirms the same behaviour in BMLoadNBody well, at least i learned something today…
day 9
debugging
so, about them nullptrs? now, i have a guess that somehow the script.cstr() isnt making it through, but time to fire up seer again and see whats happening… ive spent some time narrowing down where our data goes to die. it makes it many many functions in before going to die in a call to JSCallFree() on line 32264 of quickjs.c…
it enters there because it is apparently tagged as bytecode. is that invalid or unsupported for quickjs at this point? so the question is… why? why does it get thrown out if its bytecode by that point? there is a possibility from the looks of it to allow me to actually get something out of it. i need to figure out how to set flags to JSEVALFLAGCOMPILEONLY, which will return something other than a null object. flags is an arg to _JSEvalInternal()
i think this means i need to pass something other than 0 as the last param to JSEval() ah, it is #define'd in quickjs.h and a quick recompilation aaand nothings fixed! :)
back to the debugger… and there is no symbols… oh right.. spidermonkey… turning on debugging symbols and a new result! "is not a function!". i think that was one of my asserts tripping.
so, we have a few flags to chose from in trying then… quickjs.h:
#define JS_EVAL_TYPE_GLOBAL (0 << 0) /* global code (default) */ #define JS_EVAL_TYPE_MODULE (1 << 0) /* module code */ #define JS_EVAL_TYPE_DIRECT (2 << 0) /* direct call (internal use) */ #define JS_EVAL_TYPE_INDIRECT (3 << 0) /* indirect call (internal use) */ #define JS_EVAL_TYPE_MASK (3 << 0)#define JS_EVAL_FLAG_STRICT (1 << 3) /* force 'strict' mode /
#define JS_EVAL_FLAG_STRIP (1 << 4) / force 'strip' mode /
/ compile but do not run. The result is an object with a
JS_TAG_FUNCTION_BYTECODE or JS_TAG_MODULE tag. It can be executed
with JS_EvalFunction(). /
#define JS_EVAL_FLAG_COMPILE_ONLY (1 << 5)
/ don't include the stack frames before this eval in the Error() backtraces */
#define JS_EVAL_FLAG_BACKTRACE_BARRIER (1 << 6)
originally it was 0, so JSEVALTYPEGLOBAL is out of the question. guess i can just iterate through each of them until one works? so, global obviously fails, module fails, direct triggers an internal assert in the library, as does indirect. the other flags i dont believe are worth checking. however that comment on compile only is interesting. perhaps there is something in the docs for this?
so, once again, from the quickjs embedding example, the code looks like this:
//https://github.com/drorgl/quickjs_embedding/blob/main/src/main.c#L74 const char *expr = "function transform(a,b){return (a^2/Math.sin(2*Math.PI/b))-a/2;}"; JSValue r = JS_Eval(ctx, expr, strlen(expr), "", 0); if (JS_IsException(r)) { printf("Error Evaluating Function\r\n"); }printf("retrieving function\r\n");
JSValue global = JS_GetGlobalObject(ctx);
JSValue func = JS_GetPropertyStr(ctx, global, "transform");
and by comparison, quickjs-benchmarks.cpp: (our code)
std::string script = readfromfile("nbody.js");//actually load the file, we dont want to measure this since its our code. JSValue r = JS_Eval(ctx, script.c_str(), strlen(script.c_str()),"",JS_EVAL_TYPE_INDIRECT ); if (JS_IsException(r)) { printf("Error Evaluating Function\r\n"); } //grab the function from the script so we can call it. i didnt put this part in the measurements for v8, so its not fair to have it measured //here either JSValue global = JS_GetGlobalObject(ctx); JSValue func = JS_GetPropertyStr(ctx, global, "run"); if (JS_IsException(func)) { printf("Error Retrieving function\r\n"); }if (!JS_IsFunction(ctx, func))
{
printf("is not function!\r\n");
}
// printf("func: %s",func);
assert(JS_IsFunction(ctx,func));
for(auto _ : state){
//finally, getting the function, and running it.
std::string s = std::to_string(state.range(0));
char arg[s.length()];
memset(arg,0,sizeof(arg));
strcpy(arg,s.c_str());
JSValue args[1];
// args[0] = JS_NewInt64(ctx, state.range(0));
// args[0] = JS_NewString(ctx, std::to_string(state.range(0)).c_str());
args[0].u.ptr = &arg;
// args[0].tag = JS_TAG_STRING;
// args[0].u.int32 = s.length();
args[0] = JS_NewString(ctx,arg);
JSValue res = JS_Call(ctx,func,global,1,args);
if(JS_IsException(res)){
// JSValue err = JS_GetException(ctx);
// JSValue text = JS_ToString(ctx,res);//wow this sucks, you just cant print shit nowadays...
printf("error running simulation! %s\n",res.u.ptr);
printf("error running simulation! %s\n",args[0].u.ptr);}
JS_FreeValue(ctx,args[1]);
JS_FreeValue(ctx,res);
}
obviously, some parts are identical to the example code, for good reason too as example code is expected to work. unless im blind or something, the relevant parts of the code, except where i changed the flags in testing are identical. what could possibly be the issue here? i also noticed, staring at this that there might be a potential off-by-one error here. arg[s.length()]; might need to be arg[s.length() + 1]; as strings need to be null terminated. im going to reset the flag to 0 and try that. and back to "error running simulation!" res.u.ptr is null, and args[0].u.ptr is blank. just as before! im beginning to wonder if this code is just wrong. let me look again to see if there is anything official. wow i forgot how much the official docs say basically jack shit. https://bellard.org/quickjs/quickjs.html#Script-evaluation like, wow. this is useless! "use JSEval()" yeah i got that already. no details on with what to use it with, no example code, nothing. well, i at least found some more example code…
its short enough that i will just include the entire thing verbatim here in the notes:
#include <stddef.h> #include <stdio.h> #include <string.h>#include <quickjs.h>
int main(void) {
JSRuntime *rt = JS_NewRuntime();
JSContext *ctx = JS_NewContext(rt);char *const fooCode = "function foo(x, y) { return x + y; }";
if (JS_IsException(JS_Eval(ctx, fooCode, strlen(fooCode), "<input>", JS_EVAL_FLAG_STRICT))) {
JS_FreeContext(ctx);
JS_FreeRuntime(rt);
return -1;
}JSValue global = JS_GetGlobalObject(ctx);
JSValue foo = JS_GetPropertyStr(ctx, global, "foo");
JSValue argv[] = { JS_NewInt32(ctx, 5), JS_NewInt32(ctx, 3) };
JSValue jsResult = JS_Call(ctx, foo, global, sizeof(argv) / sizeof(JSValue), argv);
int32_t result;
JS_ToInt32(ctx, &result, jsResult);
printf("Result: %d\n", result);JSValue used[] = { jsResult, argv[1], argv[0], foo, global };
for (int i = 0; i < sizeof(used) / sizeof(JSValue); ++i) {
JS_FreeValue(ctx, used[i]);
}JS_FreeContext(ctx);
JS_FreeRuntime(rt);
return 0;
}
so, this differs in a few ways, notably in how argv is declared. and the final JSCall() args. maybe thats the problem..? lets try that ol sizeof dance… no changes… changing from passing args the old way to the way this one does it proved interesting, also changing to using int32 instead of int64 passing… segfault! the segfault was a bad printf statement. (the second one). i just disabled it. still errors out though.
interestingly, by the time the function is called, everything going into JSCall() is fine. ctx is fine, func is fine, global is fine, the args are fine.. everything. so why doesnt it do anything? lets step into JSCall() and see if we cant see anything… so during execution, it hits an exception, i didnt quite catch what one it is though…
ctx->current_exception:u = {int32 = 1433381520, float64 = 4.6355706238202789e-310, ptr = 0x5555556faa90}, tag = -1
this is the value of the exception when it is first made.
the function is planning on returning this:
ret_val:u = {int32 = 0, float64 = 0, ptr = 0x0}, tag = 6
so it jumps to the exception handler from line 16345 which is part of the switch statement for handling opcodes. specifically: quickjs.c:
CASE(OP_get_var): { JSValue val; JSAtom atom; atom = get_u32(pc); pc += 4;val = JS_GetGlobalVar<span style="color: #bc6ec5;">(</span>ctx, atom, opcode - OP_get_var_undef<span style="color: #bc6ec5;">)</span>; <span style="color: #4f97d7; font-weight: bold;">if</span> <span style="color: #bc6ec5;">(</span>unlikely<span style="color: #2d9574;">(</span>JS_IsException<span style="color: #67b11d;">(</span>val<span style="color: #67b11d;">)</span><span style="color: #2d9574;">)</span><span style="color: #bc6ec5;">)</span> <span style="color: #4f97d7; font-weight: bold;">goto</span> <span style="color: #a45bad;">exception</span>; *sp++ = val; <span style="color: #4f97d7;">}</span> BREAK;
looks like JSGetGlobalVar() is our culprit. lets rewind the debugger and step on in…
quickjs.c:
static JSValue JS_GetGlobalVar(JSContext *ctx, JSAtom prop, BOOL throw_ref_error) { JSObject *p; JSShapeProperty *prs; JSProperty *pr;<span style="color: #2aa1ae; background-color: #292e34;">/* </span><span style="color: #2aa1ae; background-color: #292e34;">no exotic behavior is possible in global_var_obj</span><span style="color: #2aa1ae; background-color: #292e34;"> */</span> p = JS_VALUE_GET_OBJ<span style="color: #bc6ec5;">(</span>ctx->global_var_obj<span style="color: #bc6ec5;">)</span>; prs = find_own_property<span style="color: #bc6ec5;">(</span>&pr, p, prop<span style="color: #bc6ec5;">)</span>; <span style="color: #4f97d7; font-weight: bold;">if</span> <span style="color: #bc6ec5;">(</span>prs<span style="color: #bc6ec5;">)</span> <span style="color: #bc6ec5;">{</span> <span style="color: #2aa1ae; background-color: #292e34;">/* </span><span style="color: #dc752f; background-color: #292e34; font-weight: bold;">XXX</span><span style="color: #2aa1ae; background-color: #292e34;">: should handle JS_PROP_TMASK properties</span><span style="color: #2aa1ae; background-color: #292e34;"> */</span> <span style="color: #4f97d7; font-weight: bold;">if</span> <span style="color: #2d9574;">(</span>unlikely<span style="color: #67b11d;">(</span>JS_IsUninitialized<span style="color: #b1951d;">(</span>pr->u.value<span style="color: #b1951d;">)</span><span style="color: #67b11d;">)</span><span style="color: #2d9574;">)</span> <span style="color: #4f97d7; font-weight: bold;">return</span> JS_ThrowReferenceErrorUninitialized<span style="color: #2d9574;">(</span>ctx, prs->atom<span style="color: #2d9574;">)</span>; <span style="color: #4f97d7; font-weight: bold;">return</span> JS_DupValue<span style="color: #2d9574;">(</span>ctx, pr->u.value<span style="color: #2d9574;">)</span>; <span style="color: #bc6ec5;">}</span> <span style="color: #4f97d7; font-weight: bold;">return</span> JS_GetPropertyInternal<span style="color: #bc6ec5;">(</span>ctx, ctx->global_obj, prop, ctx->global_obj, throw_ref_error<span style="color: #bc6ec5;">)</span>;}
so, it fails the if statement, and by the end of it the variables it has look like this:
ctx:0x555555705bf0 p:0x555555710e80 prop:418 throw_ref_error:1 prs:0x0 pr:0x0
ctx is just the context so no need to look into there. *p however, apparently a "JSObject", is a very large data structure. so, lets think here, the only happy path is if the first if passes and the second if fails. for the first if to pass, prs must not be null. which means its time to rewind again and go into findownproperty() to see why it isnt getting anything
day 10
the final day of the project. lets see what can be done.
debugging
i still have the debugger open from yesterday so im picking up exactly where i left off. jumping into findownproperty at line 4395 quickjs.c
static force_inline JSShapeProperty *find_own_property(JSProperty **ppr, JSObject *p, JSAtom atom) { JSShape *sh; JSShapeProperty *pr, *prop; intptr_t h; sh = p->shape; h = (uintptr_t)atom & sh->prop_hash_mask; h = sh->prop_hash_end[-h - 1]; prop = get_shape_prop(sh); while (h) { pr = &prop[h - 1]; if (likely(pr->atom == atom)) { *ppr = &p->prop[h - 1]; /* the compiler should be able to assume that pr != NULL here */ return pr; } h = pr->hash_next; } *ppr = NULL; return NULL; }
so, from what i can see, it just instantly skips the loop and returns null. meaning h is 0. this is confirmed as sh->prophashend is 1 apparently. this may mean i will have to go further back up and find out why it is 1. clearly given this loop, it is meant to be more than 1. by how much i dont know, but i assume not that much given the likely() which i assume is a compiler hint or something. so, where does *p->shape->prophashend get set? also what is that weird index dance they do when initializing h for the first time? to me this means p was already wrong and the issue is not in this function. looking back up the call stack, we see that p is set in JSGetGlobalVar, which we were looking at yesterday.
/* no exotic behavior is possible in global_var_obj */ p = JS_VALUE_GET_OBJ(ctx->global_var_obj);
hah, funny comment. sure thing buddy, sure. well. i guess that sets our next destination… JSVALUEGETOBJ() which is proving to be a real problem, as the debugger really does not want to step into that. ive tried like 5 times now… is it a macro? let me just search for the function and see if i find anything.
rg -F "JS_VALUE_GET_OBJ(" quickjs.h 250:#define JS_VALUE_GET_OBJ(v) ((JSObject *)JS_VALUE_GET_PTR(v))
so, it IS a macro! well, lets dig deeper, maybe we can expand this out to see what the hell it really does.
rg -F "JS_VALUE_GET_PTR(" quickjs.h 111:#define JS_VALUE_GET_PTR(v) (void *)((intptr_t)(v) & ~0xf) 139:#define JS_VALUE_GET_PTR(v) (void *)(intptr_t)(v) 216:#define JS_VALUE_GET_PTR(v) ((v).u.ptr) 250:#define JS_VALUE_GET_OBJ(v) ((JSObject *)JS_VALUE_GET_PTR(v)) 251:#define JS_VALUE_GET_STRING(v) ((JSString *)JS_VALUE_GET_PTR(v))
so we see several definitions for it here… one does a weird bitwise and with the compliment of 15? and the other just does a cast to intptrt to void * now, looking at those bitwise operations, it loks to me like a bitmask for cutting off the least significant nybble. since this is pointer stuff, im going to assume that means it is for the purposes of memory alignment to every 16 bytes. ultimately though, we have a problem here, because for some unknowable reason, JSVALUEGETPTR(v) has 3 separate definitions, so i have no way of knowing which one is even being used here. debugger, telll me what i need to know, i command thee!
ctx->global_var_obj:u = {int32 = 1433472640, float64 = 4.6355706283222051e-310, ptr = 0x555555710e80}, tag = -1
and here we can see, it has got to be v.u.ptr as that is the only one that makes sense here. which means that it is casting the pointer to a JSObject* however, if this is what p is, we may have a bigger problem, as this is supposed to be the global variable object. something must have gotten fucked up much earlier. maybe i should look at something else, what the hell do these tag numbers mean?
quickjs.c does contain a tag enum, maybe its this..?
enum { /* classid tag */ /* union usage | properties */ JS_CLASS_OBJECT = 1, /* must be first */ JS_CLASS_ARRAY, /* u.array | length */ JS_CLASS_ERROR, JS_CLASS_NUMBER, /* u.object_data */ JS_CLASS_STRING, /* u.object_data */ JS_CLASS_BOOLEAN, /* u.object_data */ JS_CLASS_SYMBOL, /* u.object_data */ JS_CLASS_ARGUMENTS, /* u.array | length */ JS_CLASS_MAPPED_ARGUMENTS, /* | length */ JS_CLASS_DATE, /* u.object_data */ JS_CLASS_MODULE_NS, JS_CLASS_C_FUNCTION, /* u.cfunc */ JS_CLASS_BYTECODE_FUNCTION, /* u.func */ JS_CLASS_BOUND_FUNCTION, /* u.bound_function */ JS_CLASS_C_FUNCTION_DATA, /* u.c_function_data_record */ JS_CLASS_GENERATOR_FUNCTION, /* u.func */ JS_CLASS_FOR_IN_ITERATOR, /* u.for_in_iterator */ JS_CLASS_REGEXP, /* u.regexp */ JS_CLASS_ARRAY_BUFFER, /* u.array_buffer */ JS_CLASS_SHARED_ARRAY_BUFFER, /* u.array_buffer */ JS_CLASS_UINT8C_ARRAY, /* u.array (typed_array) */ JS_CLASS_INT8_ARRAY, /* u.array (typed_array) */ JS_CLASS_UINT8_ARRAY, /* u.array (typed_array) */ JS_CLASS_INT16_ARRAY, /* u.array (typed_array) */ JS_CLASS_UINT16_ARRAY, /* u.array (typed_array) */ JS_CLASS_INT32_ARRAY, /* u.array (typed_array) */ JS_CLASS_UINT32_ARRAY, /* u.array (typed_array) */ #ifdef CONFIG_BIGNUM JS_CLASS_BIG_INT64_ARRAY, /* u.array (typed_array) */ JS_CLASS_BIG_UINT64_ARRAY, /* u.array (typed_array) */ #endif JS_CLASS_FLOAT32_ARRAY, /* u.array (typed_array) */ JS_CLASS_FLOAT64_ARRAY, /* u.array (typed_array) */ JS_CLASS_DATAVIEW, /* u.typed_array */ #ifdef CONFIG_BIGNUM JS_CLASS_BIG_INT, /* u.object_data */ JS_CLASS_BIG_FLOAT, /* u.object_data */ JS_CLASS_FLOAT_ENV, /* u.float_env */ JS_CLASS_BIG_DECIMAL, /* u.object_data */ JS_CLASS_OPERATOR_SET, /* u.operator_set */ #endif JS_CLASS_MAP, /* u.map_state */ JS_CLASS_SET, /* u.map_state */ JS_CLASS_WEAKMAP, /* u.map_state */ JS_CLASS_WEAKSET, /* u.map_state */ JS_CLASS_MAP_ITERATOR, /* u.map_iterator_data */ JS_CLASS_SET_ITERATOR, /* u.map_iterator_data */ JS_CLASS_ARRAY_ITERATOR, /* u.array_iterator_data */ JS_CLASS_STRING_ITERATOR, /* u.array_iterator_data */ JS_CLASS_REGEXP_STRING_ITERATOR, /* u.regexp_string_iterator_data */ JS_CLASS_GENERATOR, /* u.generator_data */ JS_CLASS_PROXY, /* u.proxy_data */ JS_CLASS_PROMISE, /* u.promise_data */ JS_CLASS_PROMISE_RESOLVE_FUNCTION, /* u.promise_function_data */ JS_CLASS_PROMISE_REJECT_FUNCTION, /* u.promise_function_data */ JS_CLASS_ASYNC_FUNCTION, /* u.func */ JS_CLASS_ASYNC_FUNCTION_RESOLVE, /* u.async_function_data */ JS_CLASS_ASYNC_FUNCTION_REJECT, /* u.async_function_data */ JS_CLASS_ASYNC_FROM_SYNC_ITERATOR, /* u.async_from_sync_iterator_data */ JS_CLASS_ASYNC_GENERATOR_FUNCTION, /* u.func */ JS_CLASS_ASYNC_GENERATOR, /* u.async_generator_data */<span style="color: #7590db;">JS_CLASS_INIT_COUNT</span>, <span style="color: #2aa1ae; background-color: #292e34;">/* </span><span style="color: #2aa1ae; background-color: #292e34;">last entry for predefined classes</span><span style="color: #2aa1ae; background-color: #292e34;"> */</span>};
a longboi. now, ive never had a need to use enums in any c/cpp programming before, so il have to look up if this is true or not, but im guessing each successive enum is 1 larger than the previous. given that it starts at 1, im assuming each one after that is a following number… looking it up it appears my intuition is correct. why then, would a value be set to -1??? and the final return from everything is tag 6, which looking at this is boolean???
there is another tag enum defined in quickjs.h
enum { /* all tags with a reference count are negative */ JS_TAG_FIRST = -11, /* first negative tag */ JS_TAG_BIG_DECIMAL = -11, JS_TAG_BIG_INT = -10, JS_TAG_BIG_FLOAT = -9, JS_TAG_SYMBOL = -8, JS_TAG_STRING = -7, JS_TAG_MODULE = -3, /* used internally */ JS_TAG_FUNCTION_BYTECODE = -2, /* used internally */ JS_TAG_OBJECT = -1,<span style="color: #7590db;">JS_TAG_INT</span> = <span style="color: #a45bad;">0</span>, <span style="color: #7590db;">JS_TAG_BOOL</span> = <span style="color: #a45bad;">1</span>, <span style="color: #7590db;">JS_TAG_NULL</span> = <span style="color: #a45bad;">2</span>, <span style="color: #7590db;">JS_TAG_UNDEFINED</span> = <span style="color: #a45bad;">3</span>, <span style="color: #7590db;">JS_TAG_UNINITIALIZED</span> = <span style="color: #a45bad;">4</span>, <span style="color: #7590db;">JS_TAG_CATCH_OFFSET</span> = <span style="color: #a45bad;">5</span>, <span style="color: #7590db;">JS_TAG_EXCEPTION</span> = <span style="color: #a45bad;">6</span>, <span style="color: #7590db;">JS_TAG_FLOAT64</span> = <span style="color: #a45bad;">7</span>, <span style="color: #2aa1ae; background-color: #292e34;">/* </span><span style="color: #2aa1ae; background-color: #292e34;">any larger tag is FLOAT64 if JS_NAN_BOXING</span><span style="color: #2aa1ae; background-color: #292e34;"> */</span>};
based on this, -1 is an object, and 6 is an exception. this makes much more sense in our context.
ok. well, that still leaves the mystery of why everything got fucked up… was it pre fucked before it ever reached the call? lets restart the program and break on the call.. the debugger is choking on something, im not sure what exactly, but it is very unhappy. ok with that settled. now i gotta look at whats up with ctx lets look at ctx->globalvarobj.u.ptr and the debugger wont let me. fun must be why they used macros in the code. i cant dereference ptr on account of it being a void, but i also cant seem to tell the debugger to treat it as a JSObject. oh there we go, thats some ugly ass casting: *(JSObject *)(ctx->globalvarobj).u.ptr
well, it wasnt even of any help. in any case…
/* before JS_Eval()*/ *(JSObject *)(ctx->global_var_obj).u.ptr:{header = {ref_count = 1, gc_obj_type = JS_GC_OBJ_TYPE_JS_OBJECT, mark = 0 '\\000', dummy1 = 1 '\\001', dummy2 = 1, link = {prev = 0x55555570c168, next = 0x5555557254e8}}, {__gc_ref_count = 1, __gc_mark = 0 '\\000', extensible = 1 '\\001', free_mark = 0 '\\000', is_exotic = 0 '\\000', fast_array = 0 '\\000', is_constructor = 0 '\\000', is_uncatchable_error = 0 '\\000', is_class = 0 '\\000', tmp_mark = 0 '\\000', class_id = 1}}, shape = 0x5555557188c0, prop = 0x5555556edec0, first_weak_ref = 0x0, u = {opaque = 0x0, bound_function = 0x0, c_function_data_record = 0x0, for_in_iterator = 0x0, array_buffer = 0x0, typed_array = 0x0, map_state = 0x0, map_iterator_data = 0x0, array_iterator_data = 0x0, regexp_string_iterator_data = 0x0, generator_data = 0x0, proxy_data = 0x0, promise_data = 0x0, promise_function_data = 0x0, async_function_data = 0x0, async_from_sync_iterator_data = 0x0, async_generator_data = 0x0, func = {function_bytecode = 0x0, var_refs = 0x81519b7bd5, home_object = 0x50}, cfunc = {c_function = {generic = 0x0, generic_magic = 0x0, constructor = 0x0, constructor_magic = 0x0, constructor_or_func = 0x0, f_f = 0x0, f_f_f = 0x0, getter = 0x0, setter = 0x0, getter_magic = 0x0, setter_magic = 0x0, iterator_next = 0x0}, length = 213 '\\325', cproto = 123 '{', magic = 20891}, array = {u1 = {size = 0, typed_array = 0x0}, u = {values = 0x81519b7bd5, ptr = 0x81519b7bd5, int8_ptr = 0x81519b7bd5 <error: Cannot access memory at address 0x81519b7bd5>, uint8_ptr = 0x81519b7bd5 <error: Cannot access memory at address 0x81519b7bd5>, int16_ptr = 0x81519b7bd5, uint16_ptr = 0x81519b7bd5, int32_ptr = 0x81519b7bd5, uint32_ptr = 0x81519b7bd5, int64_ptr = 0x81519b7bd5, uint64_ptr = 0x81519b7bd5, float_ptr = 0x81519b7bd5, double_ptr = 0x81519b7bd5}, count = 80}, regexp = {pattern = 0x0, bytecode = 0x81519b7bd5}, object_data = {u = {int32 = 0, float64 = 0, ptr = 0x0}, tag = 555419925461}}/* after JS_Eval()*/
*(JSObject *)(ctx->global_var_obj).u.ptr:{header = {ref_count = 1, gc_obj_type = JS_GC_OBJ_TYPE_JS_OBJECT, mark = 0 '\000', dummy1 = 1 '\001', dummy2 = 1, link = {prev = 0x55555570c168, next = 0x5555557254e8}}, {__gc_ref_count = 1, __gc_mark = 0 '\000', extensible = 1 '\001', free_mark = 0 '\000', is_exotic = 0 '\000', fast_array = 0 '\000', is_constructor = 0 '\000', is_uncatchable_error = 0 '\000', is_class = 0 '\000', tmp_mark = 0 '\000', class_id = 1}}, shape = 0x5555557188c0, prop = 0x5555556edec0, first_weak_ref = 0x0, u = {opaque = 0x0, bound_function = 0x0, c_function_data_record = 0x0, for_in_iterator = 0x0, array_buffer = 0x0, typed_array = 0x0, map_state = 0x0, map_iterator_data = 0x0, array_iterator_data = 0x0, regexp_string_iterator_data = 0x0, generator_data = 0x0, proxy_data = 0x0, promise_data = 0x0, promise_function_data = 0x0, async_function_data = 0x0, async_from_sync_iterator_data = 0x0, async_generator_data = 0x0, func = {function_bytecode = 0x0, var_refs = 0x81519b7bd5, home_object = 0x50}, cfunc = {c_function = {generic = 0x0, generic_magic = 0x0, constructor = 0x0, constructor_magic = 0x0, constructor_or_func = 0x0, f_f = 0x0, f_f_f = 0x0, getter = 0x0, setter = 0x0, getter_magic = 0x0, setter_magic = 0x0, iterator_next = 0x0}, length = 213 '\325', cproto = 123 '{', magic = 20891}, array = {u1 = {size = 0, typed_array = 0x0}, u = {values = 0x81519b7bd5, ptr = 0x81519b7bd5, int8_ptr = 0x81519b7bd5 <error: Cannot access memory at address 0x81519b7bd5>, uint8_ptr = 0x81519b7bd5 <error: Cannot access memory at address 0x81519b7bd5>, int16_ptr = 0x81519b7bd5, uint16_ptr = 0x81519b7bd5, int32_ptr = 0x81519b7bd5, uint32_ptr = 0x81519b7bd5, int64_ptr = 0x81519b7bd5, uint64_ptr = 0x81519b7bd5, float_ptr = 0x81519b7bd5, double_ptr = 0x81519b7bd5}, count = 80}, regexp = {pattern = 0x0, bytecode = 0x81519b7bd5}, object_data = {u = {int32 = 0, float64 = 0, ptr = 0x0}, tag = 555419925461}}
if there were any changes at all, they were not directly in here, but could be in one of the plethora of pointers that is contained within here.
conclusion
well, as this is getting nowhere, i might as well halt here as i have other things to get to today. i still have to get today's grand finale post out, and also see about submitting a PR, or hosting this myself as a fork so others can dick around with it… and so with that, im calling that a month. i was kinda wishing i could have gotten a few more interesting graphs out, but the one i did will just have to do i suppose. but hey, on the bright hand side, at least ive figured out how to get the org-mode exporter to not throw up an unusable ToC over the export!
after all the learning with n body sims i did recently, i might try messing around in godot to see if i cant make a tiny little space sim game. or, i might try messing around with boost, since that came up and thanks to this project, ive found out ways that ought to make it easy to get going.
