Tiddlywiki.files - directories - Error: Cannot create a string longer than 0x1fffffe8 characters

Dominik_P · March 11, 2025, 3:39pm

hello tw-community!

i’m trying to:

import a folder structure with files (currently testing with 23,9 GiB
2 123 files, 135 sub-folders) - mainly pictures audios videos pdfs located inside subdirectories used for tagging and topic-structuring (i create a .tid file for each directory as folder-reference)
reference those external media files using either _canonical_uri for pictures or custom field “src” for embedding files so the wiki doesn’t get too big over time…
with the help of tiddlywiki.files…

it works fine with “folders” (.tid reference) and images, eg:

{
   "directories": [
        {
            "path": "../../../../files/",
            "filesRegExp": "^.*\\.(?:jpg|jpeg|png|gif)$",
            "isTiddlerFile": false,
            "isEditableFile": true,
	        "searchSubdirectories": true,
            "fields": {
                "title": {"source": "basename-uri-decoded"},
                "created": {"source": "created"},
                "modified": {"source": "modified"},
                "type": "image/jpeg",
                "tags": {"source": "lowestofsubdirectories"},
                "text": "",
                "_canonical_uri": { "source": "filepath", "prefix": "files/" }
            }
        },
        {
            "path": "../../../../files/",
            "filesRegExp": "^.*\\.(?:tid)$",
            "isTiddlerFile": false,
            "isEditableFile": true,
	        "searchSubdirectories": true,
            "fields": {
                "title": {"source": "basename-uri-decoded"},
                "created": {"source": "created"},
                "modified": {"source": "modified"},
                "tags": {"source": "lowestofsubdirectories"},
                "parent": {"source": "lowestofsubdirectories"}
            }
        }
    ]
}

but once i add mp4 files to the mix:

        {
            "path": "../../../../files/",
            "filesRegExp": "^.*\\.(?:mp4)$",
            "isTiddlerFile": false,
            "isEditableFile": false,
	        "searchSubdirectories": true,
            "fields": {
                "title": {"source": "filename"},
                "tags": {"source": "lowestofsubdirectories"},
                "text": "",
                "src": { "source": "filepath", "prefix": "files/" }
            }
        },

i get the error message when trying to run on node:

sudo node ./tiddlywiki.js Wikis/RootWiki --wsserver  <!--bob plugin - same issue with --listen -->                                                                                     1 ✘  14s  
node:buffer:685
    slice: (buf, start, end) => buf.base64Slice(start, end),
                                    ^

**Error: Cannot create a string longer than 0x1fffffe8 characters**
    at Object.slice (node:buffer:685:37)
    at Buffer.toString (node:buffer:864:14)
    at Object.readFileSync (node:fs:491:41)
    at processFile ($:/boot/boot.js:1980:14)
    at $:/boot/boot.js:2087:7
    at $tw.utils.each ($:/boot/boot.js:146:12)
    at $tw.loadTiddlersFromSpecification ($:/boot/boot.js:2069:12)
    at $tw.loadTiddlersFromPath ($:/boot/boot.js:1950:45)
    at $:/boot/boot.js:1955:40
    at $tw.utils.each ($:/boot/boot.js:146:12) {
  code: 'ERR_STRING_TOO_LONG'
}

Node.js v23.9.0

any suggestions? help appreciated

linonetwo · March 11, 2025, 4:31pm

Hi, I can only guess, try add "type": "video/mp4",
Maybe without the mime type it will try to load file content as base64 and cause error. While I think it should have guess the memetype.

And you could read the source code of boot.js:1980:14 to understand what happened, maybe you could fix a bug of the core.

Scott_Sauyet · March 11, 2025, 5:47pm

I’m quite sure that @linonetwo is on the right track. I would follow that advice.

But if the stack trace above doesn’t mean anything to you, what it’s saying is that when it tries to read a certain file and turn its contents into a string, it ran into an issue because the maximum String length in v8, and hence in Node, is 536870888 characters. Are you trying to convert a single file larger than 512MB?

Dominik_P · March 11, 2025, 6:05pm

thanks @linonetwo - i’m a fan of your tw-contributions

…
setting the mime-type

"type": "video/mp4",

didn’t solve the issue, also tried adding

"_canonical_uri": { "source": "filepath", "prefix": "files/" },

to no avail

line 1980 of boot.js reads

text = fs.readFileSync(pathname,typeInfo.encoding || "utf8"),

from this function

/*
Load all the tiddlers defined by a `tiddlywiki.files` specification file
filepath: pathname of the directory containing the specification file
*/
$tw.loadTiddlersFromSpecification = function(filepath,excludeRegExp) {
	var tiddlers = [];
	// Read the specification
	var filesInfo = $tw.utils.parseJSONSafe(fs.readFileSync(filepath + path.sep + "tiddlywiki.files","utf8"));
	// Helper to process a file
	var processFile = function(filename,isTiddlerFile,fields,isEditableFile,rootPath) {
		var extInfo = $tw.config.fileExtensionInfo[path.extname(filename)],
			type = (extInfo || {}).type || fields.type || "text/plain",
			typeInfo = $tw.config.contentTypeInfo[type] || {},
			pathname = path.resolve(filepath,filename),
			text = fs.readFileSync(pathname,typeInfo.encoding || "utf8"),
			metadata = $tw.loadMetadataForFile(pathname) || {},
			fileTiddlers;
		if(isTiddlerFile) {
			fileTiddlers = $tw.wiki.deserializeTiddlers(path.extname(pathname),text,metadata) || [];
		} else {
			fileTiddlers =  [$tw.utils.extend({text: text},metadata)];
		}
		var combinedFields = $tw.utils.extend({},fields,metadata);
		$tw.utils.each(fileTiddlers,function(tiddler) {
			$tw.utils.each(combinedFields,function(fieldInfo,name) {
				if(typeof fieldInfo === "string" || $tw.utils.isArray(fieldInfo)) {
					tiddler[name] = fieldInfo;
				} else {
					var value = tiddler[name];
					switch(fieldInfo.source) {
						case "lowestofsubdirectories":
							value = path.relative(rootPath, filename).split(path.sep).slice(-2)[0];
							break;
						case "subdirectories":
							value = path.relative(rootPath, filename).split(path.sep).slice(0, -1);
							break;
						case "filepath":
							value = path.relative(rootPath, filename).split(path.sep).join('/');
							break;
						case "filename":
							value = path.basename(filename);
							break;
						case "filename-uri-decoded":
							value = $tw.utils.decodeURIComponentSafe(path.basename(filename));
							break;
						case "basename":
							value = path.basename(filename,path.extname(filename));
							break;
						case "basename-uri-decoded":
							value = $tw.utils.decodeURIComponentSafe(path.basename(filename,path.extname(filename)));
							break;
						case "extname":
							value = path.extname(filename);
							break;
						case "created":
							value = new Date(fs.statSync(pathname).birthtime);
							break;
						case "modified":
							value = new Date(fs.statSync(pathname).mtime);
							break;
					}
					if(fieldInfo.prefix) {
						value = fieldInfo.prefix + value;
					}
					if(fieldInfo.suffix) {
						value = value + fieldInfo.suffix;
					}
					tiddler[name] = value;
				}
			});
		});
		if(isEditableFile) {
			tiddlers.push({filepath: pathname, hasMetaFile: !!metadata && !isTiddlerFile, isEditableFile: true, tiddlers: fileTiddlers});
		} else {
			tiddlers.push({tiddlers: fileTiddlers});
		}
	};
	// Helper to recursively search subdirectories
	var getAllFiles = function(dirPath, recurse, arrayOfFiles) {
		recurse = recurse || false;
		arrayOfFiles = arrayOfFiles || [];
		var files = fs.readdirSync(dirPath);
		files.forEach(function(file) {
			if(recurse && fs.statSync(dirPath + path.sep + file).isDirectory()) {
				arrayOfFiles = getAllFiles(dirPath + path.sep + file, recurse, arrayOfFiles);
			} else if(fs.statSync(dirPath + path.sep + file).isFile()){
				arrayOfFiles.push(path.join(dirPath, path.sep, file));
			}
		});
		return arrayOfFiles;
	}
	// Process the listed tiddlers
	$tw.utils.each(filesInfo.tiddlers,function(tidInfo) {
		if(tidInfo.prefix && tidInfo.suffix) {
			tidInfo.fields.text = {prefix: tidInfo.prefix,suffix: tidInfo.suffix};
		} else if(tidInfo.prefix) {
			tidInfo.fields.text = {prefix: tidInfo.prefix};
		} else if(tidInfo.suffix) {
			tidInfo.fields.text = {suffix: tidInfo.suffix};
		}
		processFile(tidInfo.file,tidInfo.isTiddlerFile,tidInfo.fields);
	});
	// Process any listed directories
	$tw.utils.each(filesInfo.directories,function(dirSpec) {
		// Read literal directories directly
		if(typeof dirSpec === "string") {
			var pathname = path.resolve(filepath,dirSpec);
			if(fs.existsSync(pathname) && fs.statSync(pathname).isDirectory()) {
				tiddlers.push.apply(tiddlers,$tw.loadTiddlersFromPath(pathname,excludeRegExp));
			}
		} else {
			// Process directory specifier
			var dirPath = path.resolve(filepath,dirSpec.path);
			if(fs.existsSync(dirPath) && fs.statSync(dirPath).isDirectory()) {
				var	files = getAllFiles(dirPath, dirSpec.searchSubdirectories),
					fileRegExp = new RegExp(dirSpec.filesRegExp || "^.*$"),
					metaRegExp = /^.*\.meta$/;
				for(var t=0; t<files.length; t++) {
					var thisPath = path.relative(filepath, files[t]),
					filename = path.basename(thisPath);
					if(filename !== "tiddlywiki.files" && !metaRegExp.test(filename) && fileRegExp.test(filename)) {
						processFile(thisPath,dirSpec.isTiddlerFile,dirSpec.fields,dirSpec.isEditableFile,dirSpec.path);
					}
				}
			} else {
				console.log("Warning: a directory in a tiddlywiki.files file does not exist.");
				console.log("dirPath: " + dirPath);
				console.log("tiddlywiki.files location: " + filepath);
			}
		}
	});
	return tiddlers;
};

no clue though, but i guessed the same that it somehow loads the file content and causes the error

also tried to run node with larger memory

--max-old-space-size=12228 //=12GB

same result

Dominik_P · March 11, 2025, 6:15pm

thanks @Scott_Sauyet

yes some mp4 files are larger than 512mb
but i do not want to load them - instead i want to store the file path inside a tiddler
and load them with iframe or video object

Dominik_P · March 11, 2025, 6:34pm

i removed all mp4 files larger than 500 mb and same error
it works with a small amount of mp4 files though …

pmario · March 11, 2025, 8:59pm

Which OS do you use. How long is your filepath. For Windows there is a limit for file path length.

Dominik_P · March 11, 2025, 9:37pm

hi @pmario
i use manjaro
getconf PATH_MAX returns
The maximum path length is 4096 characters.

Dominik_P · March 14, 2025, 6:10pm

could the cause be that a automated generated command line string is too long?
because it works with a few files but not with the entire folder? and the sheer file size or filepath is not the cause …

oeyoews · March 15, 2025, 2:32am

ChatGPT - 字符串长度超限处理 It seems to be a problem with large files.

cdruan · March 21, 2025, 7:26am

As many have already pointed out, TW crashed because it’s reading a file that’s too large. You can try modify the source file boot\boot.js by changing line 1980 to this:

text = ("_canonical_uri" in fields ? "" : fs.readFileSync(pathname,typeInfo.encoding || "utf8")),

This will instruct TW not to load the file if _canonical_uri is supplied.

jeremyruston · March 21, 2025, 2:48pm

Thank you. I’ve pushed a fix here: Fix crash loading large files · TiddlyWiki/TiddlyWiki5@55dbce1 · GitHub

Dominik_P · March 21, 2025, 9:32pm

this gives me

TypeError: Cannot use ‘in’ operator to search for ‘_canonical_uri’ in undefined

Dominik_P · March 21, 2025, 9:33pm

tried those two changes - still same error

cdruan · March 22, 2025, 7:12am

Sorry, the code ran fine in an older version of TW. Here’s the updated fix:

text = (fields && "_canonical_uri" in fields ? "" : fs.readFileSync(pathname,typeInfo.encoding || "utf8")),

Again, replace line #1980 that looks like this:

// original line# 1980:
text = fs.readFileSync(pathname,typeInfo.encoding || "utf8"),

I tested on version 5.3.6 this time and was able to prevent crashing on large files. Make sure you add "_canonical_uri " in the fields section of your tiddlywiki.files.

Hope this helps