Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(js-sdk): Add CJS-compatible WebSocket loading #1021

Closed
wants to merge 5 commits into from

Conversation

RutamBhagat
Copy link
Contributor

@RutamBhagat RutamBhagat commented Dec 28, 2024

Fixes: #1011

The build passes for both ESM and CJS

npm run build

> @mendable/firecrawl-js@1.10.1 build
> tsup

CLI Building entry: src/index.ts
CLI Using tsconfig: tsconfig.json
CLI tsup v8.2.4
CLI Using tsup config: /home/voldemort/Desktop/Code/open_source/firecrawl/apps/js-sdk/firecrawl/tsup.config.ts
CLI Target: es2022
CLI Cleaning output folder
CJS Build start
ESM Build start
ESM dist/index.js 22.21 KB
ESM ⚡️ Build success in 17ms
CJS dist/index.cjs 24.08 KB
CJS ⚡️ Build success in 17ms
DTS Build start
DTS ⚡️ Build success in 1540ms
DTS dist/index.d.cts 15.45 KB
DTS dist/index.d.ts  15.45 KB

return require('isows').WebSocket;
} catch {
try {
return import('isows').then(m => m.WebSocket);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing an await here, right? Either here or on line 964.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching that! I've updated the code to properly handle the WebSocket loading. Could you please review the changes again?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can cause a race condition. I'd prefer importing it lazily in an already-asynchronous function, like the watch crawl endpoint's function.

@RutamBhagat RutamBhagat requested a review from mogery December 30, 2024 14:56
return require('isows').WebSocket;
} catch {
try {
return import('isows').then(m => m.WebSocket);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can cause a race condition. I'd prefer importing it lazily in an already-asynchronous function, like the watch crawl endpoint's function.

@RutamBhagat RutamBhagat requested a review from mogery December 31, 2024 10:11
Copy link
Collaborator

@tomkosm tomkosm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the work you've put into this! I believe the best approach here would be to use top-level await for ESM, while keeping the current implementation for CJS. If you're open to merging this, it might be worth exploring how to configure tsup to use distinct implementations for ESM and CJS.

I really appreciate your efforts on this, and I'm happy to assist further if needed. Let me know how I can help!

this.status = "scraping";
this.data = [];

this.initialized = CrawlWatcher.loadWebSocket()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think this is good enough, cause ws is not initialized when the constructor is called but later async, so there can be a race condition.

@RutamBhagat RutamBhagat requested a review from tomkosm January 6, 2025 10:19
Copy link
Collaborator

@tomkosm tomkosm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a fan of that approach, is there no way with tsup to have a different module for cjs and esm? Also we should take into account the the import of isows might fail if the platform doesnt have ws support.

@mogery
Copy link
Member

mogery commented Jan 7, 2025

Hi Rutam, we're going a different way to fix this, so I'm closing this PR. Thanks for the work.

@mogery mogery closed this Jan 7, 2025
@RutamBhagat RutamBhagat deleted the fix-cjs-build-error branch January 8, 2025 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Firebase error continuation
3 participants