The "Pre-Built Binaries"

[richmond62]

It cam through here: BUT the recipient "over there" did not seem to have received it.

[richmond62]

How delightful, a swift reply:

I've asked about this and here is the information I have for you:

The binaries we hosted were just a cache to speed up builds. All scripts needed to build the prebuilt binaries are present in the repository in the prebuilt folder - prebuilt/build-libraries.sh. The prebuilt/build-libraries.sh script in the repository is used to build the prebuilt libraries for mac,linux,android,ios and emscripten. On Windows the folder url is prebuilt/build-libs.bat.

All source related to the libraries built with those scripts is either downloaded as part of the script - e.g. openssl for the specified version in the 'prebuilt/versions' folder; or resides in the thirdparty repo - thirdparty folder. There are no modifications which aren't in the repositories somewhere.

[tperry2x]

How delightful, a swift reply.

Yes, but could you send Heather Laine this please:

Screenshot at 2024-02-05 11-10-05.png (57.27 KiB) Viewed 373 times

In the source that is on github, it references this URL as default.
Try to go to that URL and you'll be blocked. Perhaps this will make it clear what the issue is.

Anyone trying to build the engine from source (as we are), will come up against a compile error because curl is unable to grab the files from this directory:
https://downloads.livecode.com/prebuilts/

Screenshot at 2024-02-29 14-24-19.png (13.44 KiB) Viewed 371 times

Screenshot at 2024-02-29 14-26-22.png (42.94 KiB) Viewed 370 times

So, to say that:

All source related to the libraries built with those scripts is either downloaded as part of the script

Is incorrect I'm afraid, because it can't do this if access is restricted.

[OpenXTalkPaul]

I can't be certain of what was in that ZIP file, but based on what was in the similarly named Linux pre-built files, I built the Mac versions of bunch of those libraries and uploaded them to GitHub here: https://github.com/OpenXTalk-org/OpenXT ... /mac/Debug

Thanks Paul,
Downloaded (https://downgit.cvbox.org/#/home?url=ht ... ac%2FDebug)
I will give this a go and see what I can recreate, and see it it works.

Do you also have the original files in this Debug folder, before any modifications? - just so I can restore the originals for the engine if need be?

IIRC I compiled all of these binaries using the Xcode sub-project files that get generated by the build script. But now that I think about it, I may have modified the python .gyp or something a long time ago to get it to produce those Xcode project files. I did add the Xcode files into the repo: https://github.com/OpenXTalk-org/OpenXT ... c/livecode

richmond62 wrote: ↑Thu Feb 29, 2024 7:19 am
How delightful, a swift reply.

That is pretty much what I thought the response would be, these zip files are just a cache of dependencies binaries, and they made that cache so that these dendencies sub-project builds don't have to be recompiled every time that the Engine gets built. If they didn't cache those binaries it would be a much long time compiling the Engine(s), and remember here we're really talking about potentially a bunch of different Engine(s) and other sub-projects (Externals like revXML, revZip, etc.) and the Docs stuff as well. These libraries binaries (the code archive '.a' files) get linked into one big binary along with the rest of the compiled engine code that uses the functions (symbols) in them.

They could have put the binaries in a GitHub repo though, and linked to prebuilt binaries on GitHub in the build scripts, instead of hosting them on their own server. Sure any modifications made to produce those binaries would be available with the source (somewhere), but if they did use any other libraries without any modification (and I'm not sure if they did or not) then there wouldn't be required to provide any source. It would've been very helpful to at least have had the last guaranteed working binaries along with the source that used them, or at least know exactly which version of such a library was used.

In the end, since we're likely going to be updating all of these libraries to newer versions, does it matter what was used in the past?

[OpenXTalkPaul]

The mac 'prebuilts' project is in a sub folder as seen here: https://github.com/OpenXTalk-org/OpenXT ... e/prebuilt

[tperry2x]

The mac 'prebuilts' project is in a sub folder as seen here: https://github.com/OpenXTalk-org/OpenXT ... e/prebuilt

Thank you. I have got a copy of this downloaded and will experiment with it this evening, all being well.

[OpenXTalkPaul]

I'd like to eliminate Gyp from the process.
Older versions of Gyp used Python2
https://chromium.googlesource.com/exter ... aee80e4a7f
Gyp was updated fairly recently (last year).
https://chromium.googlesource.com/exter ... 8a50baeb07

https://gyp.gsrc.io

https://gyp.gsrc.io/docs/UserDocumentat ... troduction

There's the version of Gyp I used (from 3 years ago when I first looked at compiling the Engines) in the Repo here:
https://github.com/OpenXTalk-org/OpenXT ... /pylib/gyp

[tperry2x]

I'd like to eliminate Gyp from the process.

Yes, definitely. It seems the idea of it was to provide a platform-agnostic way of configuring the build, but I'd be happy to use an alternative.

[tperry2x]

It seems to have accepted my tar.bz2 file I made earlier.
Now onto the next missing set of prebuilts:
OpenSSL-1.1.1g-mac-Universal-PIC.tar.bz2

Screen Shot 2024-02-29 at 18.11.36.png (615.28 KiB) Viewed 336 times

Also, I'm hoping someone can help here.

Code: Select all

ld: unknown option: -platform_version

Not sure how I specify -platform_version or where I specify it.

I'm also getting:

Code: Select all

linker command failed with exit code 1 (use -v to see invocation)

I'm assuming -v is verbose, but where do I specify to use the -v flag?

[OpenXTalkPaul]

It seems to have accepted my tar.bz2 file I made earlier.
Now onto the next missing set of prebuilts:
OpenSSL-1.1.1g-mac-Universal-PIC.tar.bz2


Screen Shot 2024-02-29 at 18.11.36.png

Also, I'm hoping someone can help here.

Code: Select all

ld: unknown option: -platform_version

Not sure how I specify -platform_version or where I specify it.

I'm also getting:

Code: Select all

linker command failed with exit code 1 (use -v to see invocation)

I'm assuming -v is verbose, but where do I specify to use the -v flag?

Well I'm not sure exactly how those scripts work, but I know OpenSSL comes preinstalled by macOS, and you can also build it from source, or use HomeBrew, MacPorts or similar to build with DEV option to get the code archive (.a) version of library binary. Placing that into whatever folder the script expected them to be unzipped to (and then remove the download/unzip from the shell script) I would think should resolve it. I think it's the revSecurity external is what actually uses OpenSSL on macOS. There's no CEF used in the macOS build so it doesn't need it for Bowser Widget. I'm sure WebKit uses macOS's system installed versions of OpenSSL/TLS and the like, so it's probably only used when using the revURL library functionality (which could be probably be changed to use libCURL) or for Secure Hash Algorithms (SHA) crypto keys stuff.

I'll try to upload a binary for it later tonight (I know it's night time across the pond but it's just 5PM here).

[OpenXTalkPaul]

I just checked on a workstation running macOS 13.6 (22G120) using the OpenSSL command line and got the following:

Code: Select all

pmcclernan@Mac % /usr/bin/openssl 
OpenSSL> version
LibreSSL 3.3.6

It seems that Apple is using the LibreSSL fork of OpenSSL these days, I'm not sure if that's part of our build problems (probably not).
https://superuser.com/questions/1784424 ... l-on-macos
Anyway, I'm sure we can get a binary built for OpenSSL 1.1.1g and I believe that is one of the liberally-re-licensed newer version. There was some sort of licensing conflicts with earlier versions of OpeSSL, which is why there is an exception for that in the Community Edition's GPLv3 license.

[OpenXTalkPaul]

OpenSSL 1.1.1g

OpenSSL-1.1.1g-mac-Universal-PIC.tar.bz2

I'm guessing that from the 'Universal' it means we need a 'fat' binary as in 32bit+64bit (as opposed to the current 'fat' or 'universal' binary which would be Intel 64bit + AppleSilicon) of the library binary, since at the time (July 2021) Apple Silicon architecture was still fairly newish and was not yet supported by the v9.x engine.

[OpenXTalkPaul]

Oh, I did compile this 'stub' version which I compiled via the Xcode project that Gyp made:

https://github.com/OpenXTalk-org/OpenXT ... sl_stubs.a

Could it be that it's only looking for some sort of 'stub' version of the library, meaning that that library binary doesn't actually get used at all and is only needed to satisfy some old/outdated-on-macOS dependency?

I dunno, the stub library binary is 160kb, and looking inside the library binary I see symbol names (aka function, parameters, property enums and such) for a lot of SSL stuff.

Screenshot 2024-02-29 at 5.40.04 PM.png (1.37 MiB) Viewed 303 times

The contents of the following file suggests that I'm correct about it being only being used it the 'revSecurity' external (and I'm pretty sure that LC-Mark W. told me as much when I was trying to get going on this):
https://github.com/livecode/livecode-th ... l.gyp#L152

There is this note here: https://github.com/livecode/livecode-th ... l.gyp#L202
When it's all built to binaries the revSecurity binary on macOS must be in the same folder as the Engine binary (instead of where other externals go in the IDE).

[tperry2x]

Thank you for all your detective work on this Paul, I appreciate you looking into it.
If you do have any luck with chucking all the files you think it needs into a GitHub folder, I'll tar.bz2 it again and see if it's accepted.

The thing is exactly this though: we are having to use detective work to get things to compile from the GitHub code.

It's hardly what you'd call 'open' at the moment. Nobody is going to be able to get up and running with the engine code 'as is' currently.

I think this is the last prebuilt I'm missing for Mac OS though. That and the -platform_version not looking like it's getting passed somewhere (not sure how I fix that one yet).
Can you remember Paul what you did to get around that error and needing the -v argument passed to the script? - I'd love to know at what point that needs to be edited.

Of course, going to have all this 'fun' again (and more) when I move over to trying to compile with Windows too.

[richmond62]

I'll get back to HL in about 2 hours (90 mins of kiddos first), as obviously, not a brush off, but a deft sideways movement.

I wonder why our friends are so reticent.

[tperry2x]

I wonder why our friends are so reticent.

I really don't know, but I appreciate your help too with this Richmond. I think we need to attack it from all angles possible.
Even if they want to give us a temporary login and password so we can at least download the prebuilts directory. They can delete the login after we've got what we need if they so wish.

[richmond62]

For reasons that are not obvious, the messages section of 'the other forum' is set up in a way so that images cannot be embedded, so this is all that I could do:

I am afraid that things are not quite that simple:
-

-

-
download/file.php?id=2788
-

sincerely, Richmond.

[tperry2x]

Thank you. Fingers crossed.
I'm sorry to have to keep banging on about this. Ideally, we wouldn't have to keep mentioning it, but without the required files - there's not much chance of any improvements to the engine, and we will be limited on what we can fix in the future. This is all just to remove the initial registration screen from the code - ridiculous really, but being able to compile it successfully on all platforms is my current goal. We can then make headway into improvements in the engine from then on.

monkey_drum.gif (480.73 KiB) Viewed 276 times

[richmond62]

I'm sorry to have to keep banging on about this.

No, No, No!

This IS something that we have to keep banging on about.

Part of me wonders if our friends are not violating their own open source licence.

And that picture of the Bonobo or Chimpanzee playing a bongo drum is lovely.

[richmond62]

So . . . I got a reply:

Dear Richmond,

At this point, the problem is yours to solve. We have provided all the source under GPL, what you do with it is down to you. In this case you will need to either build libraries yourselves, upload them to a server you control and change the relevant urls in the scripts - or tweak things to not fetch libraries from a server, and instead build them locally.

GPL does not require binaries to be provided, nor even a project to build for a thirdparty - it only requires that all source be provided to those who have received binaries built from it. This we have done.

I'm sorry I cannot be any more help on this topic.